Image recognition method and device

Abstract

The present application provides an image recognition method and device. The first pixel information of a first live broadcast image in a live broadcast scene is obtained, and the second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene is obtained. If the difference information between the first pixel information and the second pixel information is less than the preset difference, it is determined that the first live broadcast image is not a sensitive image. The computing resources consumed by the method of determining whether a live broadcast image is not a sensitive image in the present application are lower than the computing resources consumed by using a recognition model of a deep neural network to determine whether a live broadcast image is a sensitive image. By screening out live broadcast images that are not sensitive images from multiple live broadcast images in a live broadcast scene in a manner that consumes less computing resources, and then using the recognition model of a deep neural network to determine whether the remaining images are sensitive images, the overall computing resources consumed in the process of identifying whether the live broadcast images in the live broadcast scene are sensitive images can be saved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image recognition method and device. Background Art

[0002] With the rapid development of technology, various live broadcasts have seen explosive growth, such as shopping live broadcasts, dating live broadcasts, and game live broadcasts.

[0003] However, as live streaming becomes more and more popular and its ability to attract traffic increases, some criminals will mix illegal content into live streaming to obtain illegal profits.

[0004] For example, in a live broadcast scenario, there will be anchors hosting the live broadcast. In order to obtain illegal benefits, some anchors will perform sensitive behaviors in the live broadcast scenario, resulting in sensitive content in the live broadcast scenario. Sensitive content includes illegal content, etc. After the sensitive content is spread through live broadcast, it will pollute the network environment and disrupt social order. Summary of the invention

[0005] The present application shows an image recognition method and device.

[0006] In a first aspect, the present application provides an image recognition method, the method comprising:

[0007] Acquire first pixel information of a first live broadcast image in a live broadcast scene, and acquire second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene;

[0008] Acquire difference information between the first pixel information and the second pixel information;

[0009] When the difference information is smaller than the preset difference, it is determined that the first live image is not a sensitive image.

[0010] In an optional implementation, the first pixel information includes a pixel value of a first pixel point in the first live image, and the second pixel information includes a pixel value of a second pixel point in the second live image;

[0011] The acquiring difference information between the first pixel information and the second pixel information includes:

[0012] For any first pixel point in the first live image, obtain the pixel position of the first pixel point in the first live image, and find a second pixel point located at the pixel position in the second live image, obtain difference information between a pixel value of the first pixel point and a pixel value of the found second pixel point, and obtain difference information corresponding to the first pixel point;

[0013] The difference information between the first pixel information and the second pixel information is obtained according to the difference information respectively corresponding to each first pixel point in the first live image.

[0014] In an optional implementation, the obtaining difference information between the pixel value of the first pixel point and the pixel value of the found second pixel point includes:

[0015] Calculating the absolute value of the difference between the pixel value of the first pixel point and the pixel value of the found second pixel point;

[0016] or,

[0017] The ratio between the larger pixel value and the smaller pixel value among the pixel value of the first pixel point and the pixel value of the found second pixel point is calculated.

[0018] In an optional implementation, the acquiring the difference information between the first pixel information and the second pixel information according to the difference information respectively corresponding to each first pixel point in the first live image includes:

[0019] The difference information corresponding to each first pixel point in the first live image is summarized to obtain the difference information between the first pixel information and the second pixel information.

[0020] In an optional implementation, the second live broadcast image is multiple;

[0021] The acquiring difference information between the first pixel information and the second pixel information includes:

[0022] Obtaining difference information between the first pixel information and second pixel information of each second live broadcast image;

[0023] And, when the difference information is less than a preset difference, determining that the first live broadcast image is not a sensitive image includes:

[0024] When the difference information between the first pixel information and the second pixel information of each second live image is less than a preset difference, it is determined that the first live image is not a sensitive image.

[0025] In an optional implementation, the acquiring first pixel information of a first live broadcast image in a live broadcast scene includes:

[0026] Acquire a target position area, the target position area including a position area where an object in the live scene is located in a live image; acquire first pixel information in the target position area of ​​the first live image in the live scene;

[0027] And, the acquiring second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene includes:

[0028] A target position area is obtained, where the target position area includes a position area where an object in the live scene is located in a live image; and second pixel information in the target position area of ​​the second live image in the live scene is obtained.

[0029] In an optional implementation, the acquiring second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene includes:

[0030] Second pixel information of a second live broadcast image in the live broadcast scene that is located before the first live broadcast image and adjacent to the first live broadcast image is obtained.

[0031] In a second aspect, the present application provides an image recognition device, the device comprising:

[0032] A first acquisition module, used to acquire first pixel information of a first live broadcast image in a live broadcast scene, and a second acquisition module, used to acquire second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene;

[0033] A third acquisition module, used to acquire difference information between the first pixel information and the second pixel information;

[0034] A determination module is used to determine that the first live image is not a sensitive image when the difference information is less than a preset difference.

[0035] In an optional implementation, the first pixel information includes a pixel value of a first pixel point in the first live image, and the second pixel information includes a pixel value of a second pixel point in the second live image;

[0036] The third acquisition module includes:

[0037] A first acquisition unit is used to acquire a pixel position of any first pixel in the first live image, and a search unit is used to search for a second pixel located at the pixel position in the second live image, and a second acquisition unit is used to acquire difference information between a pixel value of the first pixel and a pixel value of the searched second pixel, to obtain difference information corresponding to the first pixel;

[0038] The third acquisition unit is used to acquire the difference information between the first pixel information and the second pixel information according to the difference information respectively corresponding to each first pixel point in the first live image.

[0039] In an optional implementation, the second acquiring unit includes:

[0040] A first calculation subunit, used to calculate the absolute value of the difference between the pixel value of the first pixel point and the pixel value of the found second pixel point;

[0041] or,

[0042] The second calculation subunit is used to calculate the ratio between the larger pixel value and the smaller pixel value of the pixel value of the first pixel point and the pixel value of the found second pixel point.

[0043] In an optional implementation, the third acquisition unit includes:

[0044] The aggregation subunit is used to aggregate the difference information corresponding to each first pixel point in the first live image to obtain the difference information between the first pixel information and the second pixel information.

[0045] In an optional implementation, the second live broadcast image is multiple;

[0046] The third acquisition module includes:

[0047] A fourth acquisition unit, configured to acquire difference information between the first pixel information and the second pixel information of each second live image;

[0048] And, the determination module includes:

[0049] A determination unit is used to determine that the first live image is not a sensitive image when the difference information between the first pixel information and the second pixel information of each second live image is less than a preset difference.

[0050] In an optional implementation, the first acquisition module includes:

[0051] A fifth acquisition unit is used to acquire a target position area, wherein the target position area includes a position area where an object in the live scene is located in the live image; a sixth acquisition unit is used to acquire first pixel information in the target position area of ​​the first live image in the live scene;

[0052] And, the second acquisition module includes:

[0053] The seventh acquisition unit is used to acquire a target position area, wherein the target position area includes a position area where an object in the live scene is located in the live image; the eighth acquisition unit is used to acquire second pixel information within the target position area of ​​the second live image in the live scene.

[0054] In an optional implementation, the second acquisition module includes:

[0055] The ninth acquisition unit is configured to acquire second pixel information of a second live broadcast image in the live broadcast scene that is located before the first live broadcast image and adjacent to the first live broadcast image.

[0056] In a third aspect, the present application shows an electronic device, which includes: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the method shown in any of the aforementioned aspects.

[0057] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a method as shown in any of the aforementioned aspects.

[0058] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform a method as shown in any of the aforementioned aspects.

[0059] Compared with the prior art, this application has the following advantages:

[0060] In the present application, the first pixel information of the first live image in the live scene is obtained, and the second pixel information of the second live image located before the first live image in the live scene is obtained. The difference information between the first pixel information and the second pixel information is obtained. When the difference information is less than the preset difference, it is determined that the first live image is not a sensitive image. The present application screens out live images that are not sensitive images from multiple live images in the live scene in a way that consumes less computing resources, and then uses the recognition model of a deep neural network to determine whether the remaining images are sensitive images. After experimental statistics, the method of determining whether a live image is not a sensitive image in the present application consumes less computing resources, and is lower than the computing resources consumed by using the recognition model of a deep neural network to determine whether a live image is a sensitive image. In this way, the number of live images identified by the recognition model of a deep neural network can be reduced, which can reduce the computational amount of the recognition model of the deep neural network, and thus can save the overall computing resources consumed in the process of identifying whether a live image in a live scene is a sensitive image. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of a scenario shown in an exemplary embodiment of the present application.

[0062] Figure 2 It is a flowchart of an image recognition method shown in an exemplary embodiment of the present application.

[0063] Figure 3 It is a structural block diagram of an image recognition device shown in an exemplary embodiment of the present application.

[0064] Figure 4 It is a structural schematic diagram of a device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0066] Among them, sensitive behaviors include illegal behaviors, etc., and illegal behaviors include some behaviors with a large range of action execution, etc. Sensitive content includes illegal content, etc., and illegal content includes some contents with behaviors with a large range of action execution, etc.

[0067] In order to purify the network environment and stabilize public security, in one way, each live image in the live scene can be monitored to see if it is a sensitive image, and whether the sensitive image contains sensitive content. If at least one live image in the live scene is determined to be a sensitive image, it means that the live scene involves sensitive behavior, and the live broadcast can be stopped in time. For example, the live broadcast platform service end can stop the anchor from continuing to broadcast, and the anchor can be punished, and the anchor can be prohibited from broadcasting again in the future, so as to avoid further pollution of the network environment and to avoid further disturbance of public security.

[0068] For any live image frame in the live scene, a recognition model based on a deep neural network can be used to identify whether the image is a sensitive image. The same is true for each live image frame in the live scene.

[0069] However, the inventors found that there are many live broadcasts in the same time period, and the number of live broadcasts is often in the thousands, tens of thousands, or hundreds of thousands, etc., and each live broadcast includes a huge number of live images. In this way, in the process of using a deep neural network recognition model to identify whether each live image in each live broadcast in the same time period is a sensitive image, the overall computational complexity is very large, and it will consume a lot of computing resources.

[0070] Thus, there is a need to save computing resources consumed in the process of identifying whether a live broadcast image in a live broadcast scene is a sensitive image.

[0071] In order to save computing resources consumed in the process of identifying whether a live image in a live scene is a sensitive image, the inventor analyzed the above method and found that: a main function of the deep neural network recognition model is to identify sensitive images among the live images in the live scene. For example, for a single live image, a main function of the deep neural network recognition model is to process the live image to determine whether it is a sensitive image. That is, assuming that a live image is actually a sensitive image, the function of the deep neural network recognition model is to identify this live image as a sensitive image.

[0072] In this way, among the multiple live images in the live scene, if it is possible to use a method that consumes less computing resources in advance to determine that some live images are most likely not sensitive images, then these live images can be removed from the multiple live images in the live scene, and then the remaining live images can be identified using a deep neural network recognition model to determine whether they are sensitive images.

[0073] For example, a method that consumes less computing resources can be used to preliminarily screen out live images that are not sensitive images from among the live images in each live broadcast within the same time period, and then the live images that are not sensitive images can be removed from among the live images in each live broadcast within the same time period. Then, the remaining live images can be identified using a deep neural network recognition model to determine whether they are sensitive images.

[0074] In this way, the number of live broadcast images recognized by the recognition model of the deep neural network can be reduced, thereby reducing the computational complexity of the recognition model of the deep neural network.

[0075] In addition, for the live images that are initially screened out as not sensitive images, they are identified using a method that consumes less computing resources rather than using a recognition model of a deep neural network. The method that consumes less computing resources consumes less computing resources than the recognition model of a deep neural network. In this way, the overall computing resources consumed in the process of identifying whether the live images in the live scene are sensitive images can be saved.

[0076] Here, an example is used to illustrate this application. Figure 1 Among the multiple live images in the live scene, a method with lower computing resource consumption is used in advance to determine the live images that are not sensitive images, so as to divide the multiple live images in the live scene into two parts, one of which is the live images that are not sensitive images, and the other is the live images that have not been determined to be sensitive images. Then, the recognition model of the deep neural network is used to recognize the other part of the images, so as to divide the other part of the images into two parts, one of which is not sensitive images, and the other is sensitive images.

[0077] Specifically, see Figure 2 , shows a flow chart of an image recognition method of the present application, the method is applied in an electronic device, and the electronic device includes a terminal or a server.

[0078] Terminals may include mobile phones, tablet computers, laptops or desktop computers, etc.

[0079] In one embodiment, the terminal may also include a terminal used by an anchor, and the anchor may use the terminal for live broadcasting.

[0080] In this embodiment, in a live broadcast scenario, the terminal can record the host's live video, which includes multiple frames of live images, and can display the live images in real time, and can also transmit the live images to the server.

[0081] The server can forward the live broadcast image to the terminals used by the audience. The server can include the server of the live broadcast platform, etc.

[0082] The audience's terminal can receive the live image and play the live image for the audience to watch.

[0083] In another embodiment, the terminal may include a terminal used by a wide audience, and the audience may use the terminal to watch a live broadcast, etc.

[0084] In this way, in a live broadcast scene, terminals (including the host's terminal and the audience's terminal, etc.) and servers (including the server of the live broadcast platform, etc.) can obtain live images in the live broadcast scene. In this way, terminals (including the host's terminal and the audience's terminal, etc.) and servers (including the server of the live broadcast platform, etc.) can execute the process of the following embodiment to determine the live images that are not sensitive images.

[0085] Among them, the method includes:

[0086] In step S101, first pixel information of a first live broadcast image in a live broadcast scene is obtained, and second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene is obtained.

[0087] In the present application, in a live broadcast scenario, the host's terminal can record a live video, which includes multiple frames of live images, and the live images can include the host. For example, at least part of the live images in the live video have the host, etc.

[0088] For the second live image frame in the live video or any live image frame after the second frame, the present application can be used to determine in advance whether the live image is not a sensitive image. If it is determined that the image is not a sensitive image, the image can be filtered out, for example, the recognition model of the deep neural network is no longer used to further identify whether the image is a sensitive image. Alternatively, if it is not determined that the image is not a sensitive image, the image may not be filtered out, that is, the recognition model of the deep neural network can be used to further identify whether the image is a sensitive image.

[0089] For the second live image frame in the live video or any live image frame after the second frame, whenever it is necessary to determine in advance whether the live image is not a sensitive image through the method of the present application, the live image can be regarded as the first live image. Therefore, in the present application, the second live image frame in the live video or any live image frame after the second frame is taken as the first live image as an example for illustration, but it does not limit the scope of protection of the present application.

[0090] In one embodiment, when obtaining the second pixel information of the second live image located before the first live image in the live scene, the second pixel information of the second live image located before the first live image and adjacent to the first live image in the live scene can be obtained. For example, the image located before the first live image and adjacent to the first live image can be determined as the second live image, and then the second pixel information of the second live image can be obtained.

[0091] Alternatively, in another embodiment, when obtaining the second pixel information of the second live image located before the first live image in the live scene, the second pixel information of the second live image located before the first live image in the live scene and spaced at least one live image frame from the first live image may be obtained. For example, a frame of image may be first selected (e.g., randomly selected, etc.) from the live images located before the first live image in the live scene and spaced at least one live image frame from the first live image as the second live image, and then the second pixel information of the second live image may be obtained.

[0092] In the present application, the first pixel information of the first live image may include the pixel value of the first pixel point in the first live image, and the second pixel information of the second live image may include the pixel value of the second pixel point in the second live image. For example, in one embodiment, the first pixel information of the first live image may include the pixel value of each first pixel point in the first live image, and the second pixel information of the second live image may include the pixel value of each second pixel point in the second live image.

[0093] The first live image may include an image acquired in real time, or may include an image acquired at a preset time interval, or may include any image that needs to be detected, etc., and the present application does not impose any restrictions on this.

[0094] The preset time interval can be set according to actual conditions, and this application does not limit this.

[0095] In step S102, difference information between the first pixel information and the second pixel information is obtained.

[0096] In one embodiment of the present application, the first live broadcast image includes multiple pixel points, which can be regarded as multiple first pixel points, and the second live broadcast image also includes multiple pixel points, which can be regarded as multiple second pixel points.

[0097] The first pixel information includes the pixel value of each first pixel point in the first live image, and the second pixel information includes the pixel value of each second pixel point in the second live image.

[0098] In this way, the difference information between the pixel values ​​of the pixels at the same position in the first live image and the second live image can be obtained; then, the difference information between the first pixel information and the second pixel information can be obtained based on the difference information between the pixel values ​​of the pixels at the same position.

[0099] For example, for any first pixel in the first live image, the pixel position of the first pixel in the first live image can be obtained (each first pixel in the first live image has its own pixel position in the first live image), and the second pixel located at the pixel position in the second live image can be searched for, and then the difference information between the pixel value of the first pixel and the pixel value of the found second pixel can be obtained to obtain the difference information corresponding to the first pixel.

[0100] Among them, in the live broadcast scene, the size of each live video in the live broadcast scene may remain unchanged, that is, the resolution of each live image in the live broadcast scene is the same. In this way, for any first pixel point in the first live broadcast image, the first pixel point has its own pixel position in the first live broadcast image, and the pixel position may be a coordinate in the first live broadcast image, etc. Since the resolution of the first live broadcast image is the same as that of the second live broadcast image, the second live broadcast image also has the pixel position, and the pixel position in the second live broadcast image also has a second pixel point, so that the second pixel point located at the pixel position in the second live broadcast image can be found.

[0101] Wherein, when obtaining the difference information between the pixel value of the first pixel point and the pixel value of the second pixel point found, the absolute value of the difference between the pixel value of the first pixel point and the pixel value of the second pixel point found can be calculated and used as the difference information between the pixel value of the first pixel point and the pixel value of the second pixel point found. Alternatively, the ratio between the larger pixel value and the smaller pixel value of the pixel value of the first pixel point and the pixel value of the second pixel point found can be calculated and used as the difference information between the pixel value of the first pixel point and the pixel value of the second pixel point found.

[0102] The same is true for each other first pixel point in the first live image.

[0103] Then, the difference information between the first pixel information and the second pixel information can be obtained according to the difference information respectively corresponding to each first pixel point in the first live image.

[0104] For example, in one embodiment, the difference information corresponding to each first pixel point in the first live image may be summarized to obtain the difference information between the first pixel information and the second pixel information.

[0105] For example, in one example, when calculating the absolute value of the difference between the pixel value of each first pixel point and the pixel value of the corresponding second pixel point, the calculated absolute values ​​can be summarized to obtain the difference information between the first pixel information and the second pixel information. For example, the calculated absolute values ​​can be summed to obtain the difference information between the first pixel information and the second pixel information. Alternatively, the calculated absolute values ​​can be summed to obtain a numerical value, and then the number of pixels in the first live image is calculated, and then the ratio between the numerical value and the number is calculated (to achieve normalization), and used as the difference information between the first pixel information and the second pixel information.

[0106] Alternatively, in another example, when calculating the ratio between the larger pixel value and the smaller pixel value of the pixel values ​​of the respective first pixel points and the pixel values ​​of the respective corresponding second pixel points, the calculated ratios can be summarized to obtain the difference information between the first pixel information and the second pixel information. For example, the calculated ratios can be summed to obtain the difference information between the first pixel information and the second pixel information. Alternatively, the calculated ratios can be summed to obtain a numerical value, and then the number of pixels in the first live image is calculated, and then the ratio between the numerical value and the number is calculated (to achieve normalization), and used as the difference information between the first pixel information and the second pixel information.

[0107] In step S103, when the difference information is smaller than the preset difference, it is determined that the first live image is not a sensitive image.

[0108] In the present application, sensitive images include: images with illegal or unlawful content, etc. Illegal or unlawful content may include sensitive content, indecent content, and non-compliant content, etc. Non-compliant content includes content that does not comply with the specifications of the live broadcast platform, etc. The specifications of the live broadcast platform can be set according to actual needs, etc., and this application does not limit this.

[0109] For example, in one example, in a live broadcast scenario, the host performed a sensitive action during the live broadcast. After performing the sensitive action, sensitive content may appear, which in turn may cause the live broadcast image to contain sensitive content. This is a violation of laws and regulations, resulting in the live broadcast image being a sensitive image.

[0110] Among them, when performing sensitive actions, the host's movements are often large, for example, the host's hands, arms, head, neck, torso, legs and feet often have large movements.

[0111] Therefore, it can be concluded that when the host's movements are large, it is possible that the host is performing sensitive movements, which results in sensitive content in the live broadcast image, and further causes the live broadcast image to be a sensitive image.

[0112] Correspondingly, when the host's movements are relatively small, the host is often not performing sensitive movements, and sensitive content caused by performing sensitive movements often does not appear in the live broadcast image, so the live broadcast image may not be a sensitive image.

[0113] Among them, when the host's movements are large, since the movements are continuous and last for a period of time, the live images in the live video will continue to change. In addition, through the analysis of a large number of historical live videos, it can be found that the degree of change of the live image is often large. For example, the difference information between the pixel information of different live images is large.

[0114] Accordingly, when the host's movements are small, since the movements are continuous and last for a period of time, the live images in the live video will continue to change. In addition, through the analysis of a large number of historical live videos, it can be found that even if the live images continue to change, the degree of change of the live images is often small. For example, the difference information between the pixel information of different live images is small.

[0115] Furthermore, when the host does not perform any action, the live image in the live video will not change continuously, and the difference information between the pixel information of different live images can be ignored (for example, the difference information is 0, etc.).

[0116] Thus, based on this principle, it is possible to determine whether a live broadcast image is not a sensitive image through the difference information between the pixel information of the two live broadcast images.

[0117] For example, when the difference information between the pixel information of the two live images is small, it can be determined that the live image with a later time sequence in the two live images is not a sensitive image.

[0118] Alternatively, when the difference information between the pixel information of the two live images is large, it may be due to the host performing a large-scale action (which may cause sensitive content to appear in the live image), etc., which leads to the large difference information between the pixel information of the two live images, or it may be due to the host performing other actions (which will not cause sensitive content to appear in the live image), etc., which leads to the large difference information between the pixel information of the two live images. Therefore, at this time, it cannot be accurately determined that the live image with a later time sequence in the two live images is not a sensitive image, and it cannot be accurately determined that the live image with a later time sequence in the two live images is a sensitive image. That is, the live image with a later time sequence in the two live images may be a sensitive image or may not be a sensitive image. In this way, it is necessary to further and more accurately identify the live image with a later time sequence in the two live images to determine whether the live image with a later time sequence in the two live images is a sensitive image, for example, using a recognition model of a deep neural network to determine whether the live image with a later time sequence in the two live images is a sensitive image.

[0119] Thus, in this step, it can be determined whether the first live broadcast image is not a sensitive image based on the difference information between the first pixel information and the second pixel information. For example, the size relationship between the "difference information between the first pixel information and the second pixel information" and the preset difference is compared.

[0120] The preset difference may be calculated based on experience in advance, and the specific value may be determined according to actual conditions, which is not limited in this application.

[0121] When the difference information between the first pixel information and the second pixel information is smaller than the preset difference, it can be determined that the first live broadcast image is not a sensitive image.

[0122] Alternatively, when the difference information between the first pixel information and the second pixel information is greater than or equal to the preset difference, it is not certain that the first live image is not a sensitive image, that is, it is impossible to accurately determine whether the first live image is not a sensitive image.

[0123] In the present application, the first pixel information of the first live image in the live scene is obtained, and the second pixel information of the second live image located before the first live image in the live scene is obtained. The difference information between the first pixel information and the second pixel information is obtained. When the difference information is less than the preset difference, it is determined that the first live image is not a sensitive image. The present application screens out live images that are not sensitive images from multiple live images in the live scene in a way that consumes less computing resources, and then uses the recognition model of a deep neural network to determine whether the remaining images are sensitive images. After experimental statistics, the method of determining whether a live image is not a sensitive image in the present application consumes less computing resources, and is lower than the computing resources consumed by using the recognition model of a deep neural network to determine whether a live image is a sensitive image. In this way, the number of live images identified by the recognition model of a deep neural network can be reduced, which can reduce the computational amount of the recognition model of the deep neural network, and thus can save the overall computing resources consumed in the process of identifying whether a live image in a live scene is a sensitive image.

[0124] In addition, the inventors have analyzed a large number of historical live videos and found that for any frame of historical sensitive images that have appeared in historical live scenes, the difference between the pixel information of the frame of historical sensitive image and "other frames of historical sensitive images that are located before the frame of historical sensitive image in the historical live scene" is less than the preset difference. In this way, there may be a situation where the pixel information of any frame of historical sensitive image that has appeared in the historical live scene is less than the preset difference. Figure 2 The illustrated embodiment may determine that a live broadcast image that is actually a sensitive image is not a sensitive image, resulting in an erroneous determination.

[0125] In this way, in order to avoid determining that a live broadcast image that is actually a sensitive image is not a sensitive image, Figure 2 On the basis of the embodiment shown, in another embodiment of the present application, the second live image is multiple. For example, the second live image may include multiple frames of continuous historical live images before the first live image, or may also include multiple frames of historical live images sampled at intervals before the first live image.

[0126] In this way, when the difference information between the first pixel information and the second pixel information is obtained, the difference information between the first pixel information and the second pixel information of each second live image can be obtained.

[0127] Furthermore, when “when the difference information is smaller than a preset difference, it is determined that the first live image is not a sensitive image”, it can be determined that the first live image is not a sensitive image when the difference information between the first pixel information and the second pixel information of each second live image is smaller than a preset difference.

[0128] Through the present application, whether the first live image is not a sensitive image is determined by using the difference information between the first pixel information of the first live image and the second pixel information of multiple second live images, thereby avoiding as much as possible the determination of a live image that is actually a sensitive image as not a sensitive image, thereby improving the accuracy of the determination.

[0129] In the present application, there is a host in the live broadcast scene. Sometimes the host will often sit and have a verbal conversation with the audience, and rarely move. After analyzing a large number of historical live broadcast videos, it can be found that if the host performs sensitive behavior, the live broadcast image involving the sensitive behavior will be a sensitive image. If the host does not perform sensitive behavior, the live broadcast image will generally not be a sensitive image.

[0130] Thus, in a common situation, whether a live broadcast image is a sensitive image is closely related to the host, and whether a live broadcast image is a sensitive image is often unrelated to content other than the host. For example, the impact of content other than the host on whether an image is a sensitive image is often small and can be ignored.

[0131] In addition, the inventors found that in the live broadcast image in the live broadcast scene, the anchor only occupies a part of the area in the live broadcast image, and there are many areas that will not be occupied by the anchor.

[0132] Thus, in acquiring the first pixel information of the first live broadcast image in the live broadcast scene, in one approach, the pixel value of each pixel point of the first live broadcast image may be acquired.

[0133] However, according to the above analysis, the pixel information of the area where the non-anchor is located in the live image in the live scene has little effect on determining whether the live image is not a sensitive image. In the case of small effect, if the pixel information of the area where the non-anchor is located in the live image in the live scene is still calculated, it will not only waste time but also waste system resources of the electronic device (such as computing resources, which may include memory and central processing unit, etc.).

[0134] Therefore, when determining whether a live broadcast image is not a sensitive image, the pixel information of the area where the non-anchor is located in the live broadcast image in the live broadcast scene has little effect on determining whether the live broadcast image is not a sensitive image. The pixel information of the area where the non-anchor is located in the live broadcast image in the live broadcast scene may not be calculated, and the pixel information of the area where the anchor is located in the live broadcast image in the live broadcast scene may be calculated, thereby reducing the amount of calculation, improving the efficiency of determining whether the live broadcast image is not a sensitive image, and saving system resources of the electronic device.

[0135] Among them, in order to determine whether the live broadcast image is not a sensitive image, the pixel information of the area where the anchor is located in the live broadcast image in the live broadcast scene may not be calculated, and the pixel information of the area where the anchor is located in the live broadcast image in the live broadcast scene may be calculated. In one embodiment of the present application, when obtaining the first pixel information of the first live broadcast image in the live broadcast scene, the target position area may be obtained, and the target position area includes the position area where the object in the live broadcast scene is located in the live broadcast image, and then the first pixel information within the target position area of ​​the first live broadcast image in the live broadcast scene is obtained.

[0136] The size of the target position area is smaller than the size of the live image, that is, the number of pixels included in the target position area is smaller than the number of pixels in the live image.

[0137] The target location area can be determined in advance based on experience, such as the location area where the objects in the live scene are often located in the live image, etc. It can be manually counted, or it can be determined after automatic object recognition of the first few frames of live images in the live scene, etc.

[0138] For example, for a certain anchor, in the historical live broadcast of the anchor, the anchor is often located in a certain position area in the live broadcast image, then the position area is the position area for the anchor, that is, the target position area for the anchor, and the binding relationship between the anchor and the position area for the anchor can be recorded in advance.

[0139] In this way, when the live broadcast scene is a scene where the anchor is broadcasting live (the anchor's identity information can be determined by face recognition of the anchor), the location area for the anchor can be obtained according to the binding relationship and used as the target location area.

[0140] In addition, when obtaining the second pixel information of the second live image in the live scene that is located before the first live image, the target position area can be obtained, the target position area includes the position area where the object in the live scene is located in the live image, and then the second pixel information within the target position area of ​​the second live image in the live scene is obtained.

[0141] Among them, objects may include characters, etc., and tasks may include anchors, etc.

[0142] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0143] Reference Figure 3 , shows a structural block diagram of an image recognition device of the present application, the device comprising:

[0144] A first acquisition module 11 is used to acquire first pixel information of a first live broadcast image in a live broadcast scene, and a second acquisition module 12 is used to acquire second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene;

[0145] A third acquisition module 13, used to acquire difference information between the first pixel information and the second pixel information;

[0146] The determination module 14 is configured to determine that the first live broadcast image is not a sensitive image when the difference information is smaller than a preset difference.

[0147] In an optional implementation, the first pixel information includes a pixel value of a first pixel point in the first live image, and the second pixel information includes a pixel value of a second pixel point in the second live image;

[0148] The third acquisition module includes:

[0149] A first acquisition unit is used to acquire a pixel position of any first pixel in the first live image, and a search unit is used to search for a second pixel located at the pixel position in the second live image, and a second acquisition unit is used to acquire difference information between a pixel value of the first pixel and a pixel value of the searched second pixel, to obtain difference information corresponding to the first pixel;

[0150] The third acquisition unit is used to acquire the difference information between the first pixel information and the second pixel information according to the difference information respectively corresponding to each first pixel point in the first live image.

[0151] In an optional implementation, the second acquiring unit includes:

[0152] A first calculation subunit, used to calculate the absolute value of the difference between the pixel value of the first pixel point and the pixel value of the found second pixel point;

[0153] or,

[0154] The second calculation subunit is used to calculate the ratio between the larger pixel value and the smaller pixel value of the pixel value of the first pixel point and the pixel value of the found second pixel point.

[0155] In an optional implementation, the third acquisition unit includes:

[0156] The aggregation subunit is used to aggregate the difference information corresponding to each first pixel point in the first live image to obtain the difference information between the first pixel information and the second pixel information.

[0157] In an optional implementation, the second live broadcast image is multiple;

[0158] The third acquisition module includes:

[0159] A fourth acquisition unit, configured to acquire difference information between the first pixel information and the second pixel information of each second live image;

[0160] And, the determination module includes:

[0161] A determination unit is used to determine that the first live image is not a sensitive image when the difference information between the first pixel information and the second pixel information of each second live image is less than a preset difference.

[0162] In an optional implementation, the first acquisition module includes:

[0163] A fifth acquisition unit is used to acquire a target position area, wherein the target position area includes a position area where an object in the live scene is located in the live image; a sixth acquisition unit is used to acquire first pixel information in the target position area of ​​the first live image in the live scene;

[0164] And, the second acquisition module includes:

[0165] The seventh acquisition unit is used to acquire a target position area, wherein the target position area includes a position area where an object in the live scene is located in the live image; the eighth acquisition unit is used to acquire second pixel information within the target position area of ​​the second live image in the live scene.

[0166] In an optional implementation, the second acquisition module includes:

[0167] The ninth acquisition unit is configured to acquire second pixel information of a second live broadcast image in the live broadcast scene that is located before the first live broadcast image and adjacent to the first live broadcast image.

[0168] In the present application, the first pixel information of the first live image in the live scene is obtained, and the second pixel information of the second live image located before the first live image in the live scene is obtained. The difference information between the first pixel information and the second pixel information is obtained. When the difference information is less than the preset difference, it is determined that the first live image is not a sensitive image. The present application screens out live images that are not sensitive images from multiple live images in the live scene in a way that consumes less computing resources, and then uses the recognition model of a deep neural network to determine whether the remaining images are sensitive images. After experimental statistics, the method of determining whether a live image is not a sensitive image in the present application consumes less computing resources, and is lower than the computing resources consumed by using the recognition model of a deep neural network to determine whether a live image is a sensitive image. In this way, the number of live images identified by the recognition model of a deep neural network can be reduced, which can reduce the computational amount of the recognition model of the deep neural network, and thus can save the overall computing resources consumed in the process of identifying whether a live image in a live scene is a sensitive image.

[0169] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0170] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, the electronic device executes one or more methods in the above embodiments. In the present application embodiment, the electronic device includes a server, a gateway, a sub-device, etc., and the sub-device is an Internet of Things device or other device.

[0171] The embodiments of the present disclosure may be implemented as an apparatus configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the apparatus may include electronic devices such as servers (clusters), terminal devices such as IoT devices, and the like.

[0172] Figure 4 An exemplary device 1300 that can be used to implement various embodiments of the present application is schematically shown.

[0173] For one embodiment, Figure 4An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the (one or more) processors 1302, a memory 1306 coupled to the control module 1304, a non-volatile memory (NVM) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0174] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can be used as a server device such as a gateway in the embodiments of the present application.

[0175] In some embodiments, the device 1300 may include one or more computer-readable media (e.g., memory 1306 or NVM / storage device 1308) having instructions 1314 and one or more processors 1302 configured to execute the instructions 1314 in combination with the one or more computer-readable media to implement a module to perform the actions of the present disclosure.

[0176] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .

[0177] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0178] The memory 1306 may be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 1306 may include a double data rate quad synchronous dynamic random access memory (DDR4 SDRAM).

[0179] For one embodiment, control module 1304 may include one or more input / output controllers to provide an interface to NVM / storage device 1308 and input / output device(s) 1310 .

[0180] For example, NVM / storage 1308 may be used to store data and / or instructions 1314. NVM / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0181] NVM / storage device 1308 may include storage resources that are physically part of the device on which apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1308 may be accessed via input / output device(s) 1310 over a network.

[0182] (One or more) input / output devices 1310 may provide an interface for the apparatus 1300 to communicate with any other appropriate device, and the input / output device 1310 may include a communication component, a phonetic component, a sensor component, etc. The network interface 1312 may provide an interface for the apparatus 1300 to communicate through one or more networks, and the apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0183] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip (SoC).

[0184] In various embodiments, the device 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0185] An embodiment of the present application provides an electronic device, comprising: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, enable the electronic device to perform one or more methods in the present application.

[0186] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0187] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0188] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of the processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable information processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable information processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable information processing terminal device, so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0191] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0192] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0193] The above is a detailed introduction to an image recognition method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image recognition method, characterized in that: The method comprises: Acquire first pixel information of a first live broadcast image in a live broadcast scene, and acquire second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene; Acquire difference information between the first pixel information and the second pixel information; When the difference information is less than a preset difference, determining that the first live broadcast image is not a sensitive image; There are multiple second live images; the second live images include multiple consecutive frames of historical live images before the first live image, or multiple frames of historical live images sampled at intervals before the first live image; The acquiring difference information between the first pixel information and the second pixel information includes: Obtaining difference information between the first pixel information and second pixel information of each second live broadcast image; And, when the difference information is less than a preset difference, determining that the first live broadcast image is not a sensitive image includes: When the difference information between the first pixel information and the second pixel information of each second live image is less than a preset difference, it is determined that the first live image is not a sensitive image.

2. The method according to claim 1, characterized in that The first pixel information includes a pixel value of a first pixel point in the first live image, and the second pixel information includes a pixel value of a second pixel point in the second live image; The acquiring difference information between the first pixel information and the second pixel information includes: For any first pixel point in the first live image, obtain the pixel position of the first pixel point in the first live image, and find a second pixel point located at the pixel position in the second live image, obtain difference information between a pixel value of the first pixel point and a pixel value of the found second pixel point, and obtain difference information corresponding to the first pixel point; The difference information between the first pixel information and the second pixel information is obtained according to the difference information respectively corresponding to each first pixel point in the first live image.

3. The method according to claim 2, characterized in that The obtaining difference information between the pixel value of the first pixel point and the pixel value of the found second pixel point includes: Calculating the absolute value of the difference between the pixel value of the first pixel point and the pixel value of the found second pixel point; or, The ratio between the larger pixel value and the smaller pixel value among the pixel value of the first pixel point and the pixel value of the found second pixel point is calculated.

4. The method according to claim 2 or 3, characterized in that: The obtaining the difference information between the first pixel information and the second pixel information according to the difference information respectively corresponding to each first pixel point in the first live image includes: The difference information corresponding to each first pixel point in the first live image is summarized to obtain the difference information between the first pixel information and the second pixel information.

5. The method according to claim 1, characterized in that The obtaining first pixel information of a first live broadcast image in the live broadcast scene includes: Acquire a target position area, the target position area including a position area where an object in the live scene is located in a live image; acquire first pixel information in the target position area of ​​the first live image in the live scene; And, the acquiring second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene includes: A target position area is obtained, where the target position area includes a position area where an object in the live scene is located in a live image; and second pixel information in the target position area of ​​the second live image in the live scene is obtained.

6. The method according to claim 1, characterized in that The acquiring second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene includes: Second pixel information of a second live broadcast image in the live broadcast scene that is located before the first live broadcast image and adjacent to the first live broadcast image is obtained.

7. An image recognition device, characterized in that: The device comprises: A first acquisition module, used to acquire first pixel information of a first live broadcast image in a live broadcast scene, and a second acquisition module, used to acquire second pixel information of a second live broadcast image located before the first live broadcast image in the live broadcast scene; A third acquisition module, used to acquire difference information between the first pixel information and the second pixel information; A determination module, configured to determine that the first live broadcast image is not a sensitive image when the difference information is less than a preset difference; There are multiple second live images; the second live images include multiple consecutive frames of historical live images before the first live image, or multiple frames of historical live images sampled at intervals before the first live image; The third acquisition module includes: A fourth acquisition unit, configured to acquire difference information between the first pixel information and the second pixel information of each second live image; And, the determination module includes: A determination unit is used to determine that the first live image is not a sensitive image when the difference information between the first pixel information and the second pixel information of each second live image is less than a preset difference.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images