Scene identification method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a scene identification method and device, electronic equipment and a medium. The method specifically comprises the following steps: determining a character category corresponding to a pixel point in a to-be-processed image; determining a first scene category corresponding to the character category according to a picture proportion corresponding to the character category; the picture proportion is the proportion of the image area of the character category relative to the total image area of the image to be processed; and correcting the first scene category according to the posture key points corresponding to the character category to obtain a second scene category corresponding to the character category. According to the embodiment of the invention, the labor cost of scene identification can be saved, the scene identification efficiency can be improved, and the accuracy and reliability of a scene identification result can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image recognition technology, and in particular to a scene recognition method, device, electronic equipment and medium. Background Art

[0002] The field of view refers to the difference in the size of the subject presented in the camera recorder due to the different distances between the camera and the subject when the focal length is constant. The field of view from near to far is: close-up, close-up, medium shot, panoramic shot, and long shot. Effectively identifying the field of view of video images can help improve the efficiency of retrieval materials on the one hand, and can also effectively enrich the expression of intelligent editing creation on the other hand.

[0003] Current scene recognition methods often rely on users' personal experience to observe and subjectively judge the scene of video images when editing videos.

[0004] However, current scene recognition methods often require users to have a certain professional foundation in editing, directing, film and television, etc., with high labor costs, low recognition efficiency and high error probability. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a scene recognition method, device, electronic device and medium, which can save the labor cost of scene recognition, improve the efficiency of scene recognition, and improve the accuracy and reliability of scene recognition results.

[0006] In a first aspect of the present invention, a scene recognition method is provided, the method comprising:

[0007] Determine the person category corresponding to the pixel point in the image to be processed;

[0008] Determine the first scene category corresponding to the person category according to the picture ratio corresponding to the person category; the picture ratio is the ratio of the image area of ​​the person category to the total image area of ​​the image to be processed;

[0009] According to the posture key points corresponding to the character category, the first scene category is corrected to obtain a second scene category corresponding to the character category.

[0010] In a second aspect of the present invention, a scene recognition device is provided, the device comprising:

[0011] A character category determination module is used to determine the character category corresponding to the pixel point in the image to be processed;

[0012] An initial scene determination module is used to determine a first scene category corresponding to the character category according to the picture ratio corresponding to the character category; the picture ratio is the ratio of the image area of ​​the character category to the total image area of ​​the image to be processed;

[0013] The scene correction module is used to correct the first scene category according to the posture key points corresponding to the character category to obtain a second scene category corresponding to the character category.

[0014] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0015] Memory, used to store computer programs;

[0016] The processor is used to implement the aforementioned method steps when executing the program stored in the memory.

[0017] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute any of the above-described methods.

[0018] In a fifth aspect of the implementation of the present invention, there is also provided a computer program product, comprising a computer program / instruction, wherein the computer program / instruction implements any of the above-mentioned methods when executed by a processor.

[0019] The scene recognition method, device, electronic device and medium provided by the embodiments of the present invention are automatically operated by a computer program. Specifically, the computer program first determines the person category corresponding to the pixel point in the image to be processed, then determines the first scene category based on the objective screen ratio of the person category in the image, and then corrects the first scene category using the posture key points corresponding to the person category.

[0020] Since the method of the embodiment of the present invention can save tedious and time-consuming manual analysis work, it can save the labor cost of scene recognition. In addition, the method makes use of the powerful computing power and fast data processing characteristics of computers, so the embodiment of the present invention can improve the efficiency of scene recognition.

[0021] Furthermore, the embodiment of the present invention first determines the first scene category by using the first picture rule of "the image area is larger when near and smaller when far away"; and then corrects the first scene category by using the second picture rule of "the presentation rules of the details of the person's posture are different under different scenes". The combination of the first picture rule and the second picture rule can comprehensively determine the scene category corresponding to the person category from two dimensions: macroscopic (screen ratio) and microscopic (posture details). Therefore, the embodiment of the present invention can improve the accuracy and reliability of the scene recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0023] Figure 1 A flowchart of a scene recognition method according to an embodiment of the present invention;

[0024] Figure 2 A flowchart of a scene recognition method according to an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of the structure of a scene recognition device according to an embodiment of the present invention;

[0026] Figure 4 The structure block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0028] The scene recognition method of the embodiment of the present invention can be used to identify the scene category corresponding to the person in the image to be processed. In one example, the scene categories include, from far to near: distant view category, panoramic category, medium panoramic category, medium view category, medium close view category, close view category, face close-up category, and eye close-up category.

[0029] The following is a detailed description of the above categories:

[0030] Long shot category: In the long shot category, the characters usually occupy a small part of the picture. Although the whole body of the characters can be shown, they are used as decorative elements. The focus is on expressing the environment, atmosphere and vast space.

[0031] Panorama category: In the panorama category, the characters usually occupy about one-third or more of the picture, which can fully display the whole body image of the characters from head to toe and the scene in which the characters are located. The characters are close to the upper and lower boundaries of the picture, which is often used to clearly express the relationship between the characters and the environment.

[0032] Medium panorama category: The medium panorama category is between panorama and medium shot. The height of the character in the picture is about one-half of the height of the picture. Compared with the panorama category, it focuses more on highlighting the main body of the character, but still retains a certain environmental background display.

[0033] Medium shot category: In the medium shot category, the character usually occupies about two-thirds of the height of the picture, mainly showing the part above the knees. The focus is on showing the character's gestures, expressions and upper body movement details, and the proportion of the environment in the picture is relatively reduced.

[0034] Medium close-up category: The medium close-up category usually shows the part of the character above the waist. The character occupies most of the picture. The height of the character usually accounts for about three-quarters of the height of the picture. The character's facial expression, eyes and other details are more prominent, and the environmental background is further weakened.

[0035] Close-up category: In the close-up category, the character usually occupies about four-fifths of the picture, mainly showing the part above the chest. The character dominates the picture, and facial details such as facial features and expressions become the focus of the picture. Environmental factors are basically just used as background. One difference between the close-up category and the medium close-up category is whether the character's chest is above the lower boundary of the picture. If it is above the lower boundary, it is a medium close-up category. In the close-up category, the character's chest is below the lower boundary of the picture, that is, it is invisible.

[0036] Face close-up category: In the face close-up category, the person's face almost occupies the entire screen, and only a very small number of environmental elements may appear in a very small part of the edge, which is mainly used to show the facial details of the person to the extreme.

[0037] Eye close-up category: In the eye close-up category, the picture is almost entirely focused on the person's eyes, and other facial elements and environmental elements are basically invisible.

[0038] The embodiment of the present invention does not limit the specific application scenarios corresponding to the scene recognition method. For example, the application scenarios of the scene recognition method may include: film and television production, image editing, security monitoring and many other application scenarios.

[0039] In the related technologies, users usually observe and subjectively judge the scene of the video image based on their personal experience. However, the related technologies often require users to have a certain professional foundation in editing, directing, film and television, etc., which has high labor costs, low recognition efficiency and high error probability.

[0040] In view of the technical problems of high labor cost, low recognition efficiency and high error probability in scene recognition in the related art, an embodiment of the present invention provides a scene recognition method, which specifically includes:

[0041] Determine the person category corresponding to the pixel point in the image to be processed;

[0042] Determine the first scene category corresponding to the above-mentioned character category according to the picture ratio corresponding to the above-mentioned character category; the above-mentioned picture ratio is the ratio of the image area of ​​the character category to the total image area of ​​the image to be processed;

[0043] According to the posture key points corresponding to the above-mentioned character category, the above-mentioned first scene category is corrected to obtain the second scene category corresponding to the above-mentioned character category.

[0044] In the field of image recognition, person categories are categories related to people that are divided based on pixel features. By analyzing the image features such as human body contours, colors, and textures displayed by pixels, it is determined whether the pixel belongs to the category of people, and the pixels representing the same person are classified into one category to distinguish different people in multiple images to be processed. In practical applications, different person categories can be distinguished based on the person category ID (identity).

[0045] The embodiment of the present invention provides a scene recognition method, which is automatically operated by a computer program. Specifically, the computer program first determines the person category corresponding to the pixel point in the image to be processed, then determines the first scene category based on the objective screen ratio of the person category in the image, and then corrects the first scene category using the posture key points corresponding to the person category.

[0046] Since the method of the embodiment of the present invention can save tedious and time-consuming manual analysis work, it can save the labor cost of scene recognition. In addition, the method makes use of the powerful computing power and fast data processing characteristics of computers, so the embodiment of the present invention can improve the efficiency of scene recognition.

[0047] Furthermore, the embodiment of the present invention first determines the first scene category by using the first picture rule of "the image area is larger when it is near and smaller when it is far away"; and then corrects the first scene category by using the second picture rule of "the presentation rules of the details of the person's posture are different under different scenes". The combination of the first picture rule and the second picture rule can comprehensively determine the scene category corresponding to the person category from two dimensions: macroscopic (screen ratio) and microscopic (posture details). Therefore, the embodiment of the present invention can improve the accuracy and reliability of the scene recognition results.

[0048] In example A, the second picture rules corresponding to the panoramic category and the distant view category are similar, both requiring the feet and the top of the head to be exposed. If only relying on the second picture rule, it is difficult to accurately distinguish the panoramic category from the distant view category because the second picture rules of the two are similar, resulting in errors in the scene recognition results. The embodiment of the present invention combines the first picture rule and the second picture rule, relying on the obvious difference in image area between the two (the image area corresponding to the panoramic category and the distant view category is different), and accurately distinguishes the panoramic category from the distant view category based on this, so it can effectively improve the accuracy and reliability of the scene recognition results.

[0049] In another example, when there are many occluded parts in the image to be processed, if the scene category is determined only according to the first picture rule, the picture ratio is prone to deviation. The embodiment of the present invention can use posture recognition technology to perform posture recognition on the image to be processed to obtain the posture key points corresponding to each character category. Even if the character is partially occluded, the posture key point technology can reasonably predict the position of the posture key points corresponding to the occluded body parts based on the position combination characteristics presented by the visible body parts and the coherence of the human body structure. On this basis, these predicted posture key points are taken into consideration and used in the correction process of the first scene category, so as to effectively improve the accuracy of scene recognition and make the scene determination result more consistent with the real shooting scene.

[0050] The embodiments of the present invention are described below through specific examples.

[0051] Reference Figure 1 , shows a flowchart of a scene recognition method according to an embodiment of the present invention, the method may specifically include the following steps:

[0052] Step 101, determining the person category corresponding to the pixel point in the image to be processed;

[0053] Step 102: determining the first scene category corresponding to the person category according to the picture ratio corresponding to the person category; the picture ratio is specifically the ratio of the image area of ​​the person category to the total image area of ​​the image to be processed;

[0054] Step 103: Correct the first scene category according to the posture key points corresponding to the character category to obtain a second scene category corresponding to the character category.

[0055] Figure 1 The illustrated method embodiment can be used to identify the scene category corresponding to a person in an image to be processed. Figure 1 The method embodiment shown can be executed by a client or a server. It can be understood that the embodiment of the present invention is for Figure 1 The specific execution subject of the illustrated method embodiment is not limited.

[0056] In step 101, the image to be processed may refer to an image to be subjected to scene recognition. The user may input the image to be processed to the client by operating the client, such as selecting the desired image on the image upload interface of the device, or directly dragging the image to the designated area. The server receives the user input image forwarded from the client through the network, or directly receives the image to be processed submitted by the user through professional software or web page on a specific image upload interface.

[0057] In an application example of the present invention, an embodiment of the present invention can obtain an image to be processed whose resolution and clarity meet set conditions from an image set.

[0058] For images with too low resolution, due to the limited detailed information contained, key elements such as the outline of the person and key points of posture may not be clearly presented, which will affect the accuracy of the subsequent judgment of the person category in the image, and is not conducive to accurate analysis of the picture ratio and scene correction based on key points of posture. Images with poor clarity also have such problems. Blurred images make it difficult to accurately extract and identify image features. By screening out images to be processed that meet the requirements of resolution and clarity, a high-quality data foundation can be provided for the subsequent scene recognition process.

[0059] The process of obtaining an image to be processed whose resolution and clarity meet set conditions from an image collection specifically includes: using a computer vision library to obtain the resolution of the image, calculating the number of image pixels, filtering out images whose resolution does not meet the first set condition, and retaining the image to be processed whose resolution meets the first set condition.

[0060] Resolution specifically includes horizontal and vertical pixel counts, such as the common 1920×1080 (horizontal pixel count × vertical pixel count), which directly reflects the size of the image and its potential ability to contain details. The number of image pixels is the product of the horizontal and vertical pixel counts, which more accurately measures the size of the image.

[0061] The first setting condition specifically includes: the number of horizontal pixels is greater than the fifth threshold, and / or the number of vertical pixels is greater than the sixth threshold, and / or the number of image pixels is greater than the seventh threshold, etc.

[0062] The process of obtaining an image to be processed whose resolution and clarity meet set conditions from an image set specifically includes: using a Laplace operator to determine the clarity of the image, filtering out images whose clarity does not meet a second set condition, and retaining images to be processed whose clarity meets the second set condition.

[0063] The Laplace operator can capture the areas where the grayscale values ​​of pixels in an image change dramatically, that is, the edges of the image, and comprehensively quantify the clarity of the image by analyzing the edge information.

[0064] Since images of different types and resolutions have significant differences in clarity, such differences are reflected in the fact that high-resolution images can often present extremely rich and delicate details, medium-resolution images have a lower detail presentation capability, and low-resolution images have a relatively limited presentation capability. Therefore, in order to more accurately and reasonably select images whose clarity meets the requirements of subsequent processing, a second setting condition is set for images in different resolution ranges to classify and screen images in the image set.

[0065] Specifically, for high-resolution images, given their strong detail-carrying capacity, in order to make these rich details clearly discernible, after using the Laplace operator to calculate the image clarity, its clarity value needs to be higher than a relatively high eighth threshold before it can be determined that the image clarity meets the requirements and then retained as the image to be processed.

[0066] For medium-resolution images, the number of pixels and the degree of detail are at an intermediate level. Accordingly, as long as the clarity value is higher than a relatively moderate ninth threshold, it is considered that its clarity can meet the basic standards required for subsequent scene recognition and other operations, and meet the conditions for entering the next processing flow.

[0067] For low-resolution images, although the overall number of pixels is small and the details that can be presented are relatively limited, they also need to have a certain bottom line of clarity. Therefore, it may only be required that the clarity value is higher than a relatively small tenth threshold to be considered as meeting the clarity standard.

[0068] From the perspective of the relationship between the threshold sizes, the eighth threshold is greater than the ninth threshold, and the ninth threshold is greater than the tenth threshold. Such settings meet the actual gradient changes required for image clarity at different resolutions, and can effectively screen various images in accordance with their own characteristics.

[0069] It should be emphasized that the fifth to tenth thresholds mentioned above can all be positive real numbers. The specific values ​​will be determined comprehensively based on the actual application scenarios, past experience and the results of a large number of tests and verifications, so that the images to be processed selected from the image collection can achieve an ideal state in terms of both resolution and clarity.

[0070] In practical applications, semantic segmentation technology can be used to determine the person category corresponding to the pixel points in the image to be processed.

[0071] Semantic segmentation technology is a technology in the field of computer vision, which aims to classify each pixel in the image to be processed into a specific semantic category. Its core task is to divide the image to be processed into multiple semantic regions, each of which has the same semantic category. This means that each pixel in the image to be processed will be assigned a semantic category, such as "person", "sky", "tree", "building", etc. It can be understood that the embodiments of the present invention do not limit the specific semantic segmentation technology.

[0072] An image to be processed may contain one or more person categories. The embodiment of the present invention may use person category identifiers to distinguish different person categories in the image to be processed. For example, person category 1 represents person 1, person category 2 represents person 2, and so on.

[0073] In step 102, the embodiment of the present invention calculates the picture ratio corresponding to the person category, and then matches the picture ratio with the picture ratio range corresponding to different scene categories in combination with the first picture rule of "the image area is larger near and smaller far", so as to determine the first scene category corresponding to the person category. In this way, the scene categories can be quickly and preliminarily divided according to the picture ratio of the person in the image to be processed, providing a basic judgment basis for subsequent scene correction and other operations.

[0074] The process of determining the screen ratio corresponding to the character category specifically includes:

[0075] First, count the number of pixels corresponding to the person category as the image area of ​​the person category. The person category corresponding to each pixel has been determined based on the semantic segmentation technology. Here, the number of pixels belonging to the same person category can be counted.

[0076] Then, the total number of pixels of the image to be processed is determined as the total image area of ​​the image to be processed; the total number of pixels may be the product of the number of horizontal pixels and the number of vertical pixels of the image to be processed;

[0077] Next, the image area of ​​the person category is divided by the total image area of ​​the image to be processed to obtain the screen ratio corresponding to the person category.

[0078] The embodiment of the present invention can search the corresponding relationship between the picture ratio range and the scene category according to the picture ratio corresponding to the character category to obtain the first scene category corresponding to the character category. The above corresponding relationship provides a quantitative standard for determining the first scene category.

[0079] Referring to Table 1, an example of the correspondence between the aspect ratio range and the scene type of the present invention is shown.

[0080] Table 1

[0081]

[0082]

[0083] The process of determining the first scene category corresponding to the above-mentioned character category according to the picture ratio corresponding to the above-mentioned character category may specifically include:

[0084] Step A1: when the picture ratio corresponding to the person category is greater than or equal to a first threshold, determining the first scene category corresponding to the person category as a close-up scene category; or

[0085] Step A2: when the picture ratio corresponding to the person category is greater than or equal to the second threshold and less than the first threshold, determining the first scene category corresponding to the person category as a medium or close scene category; or

[0086] Step A3: when the picture ratio corresponding to the person category is greater than or equal to the third threshold and less than the second threshold, determining the first scene category corresponding to the person category as a medium panoramic category; or

[0087] Step A4: when the picture ratio corresponding to the person category is greater than or equal to the fourth threshold and less than the third threshold, determining the first scene category corresponding to the person category as a panoramic category; or

[0088] Step A5: when the picture ratio corresponding to the person category is less than a fourth threshold, determining the first scene category corresponding to the person category as a distant scene category;

[0089] Among them, the first threshold is greater than the second threshold, the second threshold is greater than the third threshold, and the third threshold is greater than the fourth threshold.

[0090] In step A1, the close-up category is used to present the detailed elements of the main body of the character, such as the upper body or facial expression of the character. Because these detailed elements need to be presented in detail, the character should occupy a relatively large area ratio in the picture.

[0091] The first threshold is used as a key judgment boundary. When the screen ratio corresponding to the person category reaches or exceeds it, it intuitively indicates that the area of ​​the person in the image is large enough, which can fit the characteristic that the person in the close-up category is closer to the camera.

[0092] In step A2, the medium close-up category is in the transition interval between the close-up and the medium close-up. On the one hand, it continues the ability to show the main body of the character, and the facial expression, hand movement focus, etc. of the character can still be captured by the viewer; on the other hand, it begins to integrate certain surrounding environmental elements to expand the amount of information on the screen. Because both must be taken into account, the proportion of the character in the picture is slightly smaller than that of the pure close-up, so the second threshold value is less than the first threshold. When the picture ratio is greater than or equal to the second threshold and less than the first threshold, it means that the size of the character in the picture is just right, and the distance between the character and the lens is moderate. It will not only highlight the details of the character and ignore the environment because it is too large, nor will it be too small to blur the main body of the character, which meets the picture presentation effect that the medium close-up should have.

[0093] In step A3, the medium panorama category is used to highlight the main body of the person from the knees to the top of the head, focusing on depicting the body dynamics of the person in walking, standing and other postures.

[0094] The rich elements displayed in the medium panoramic category include the following categories: one is the character's own details; two is the architectural environment elements; three is the site facility elements; four is the surrounding group portrait elements.

[0095] In order to present these rich elements, the person is farther away from the camera lens. Following the rule of "the image area is larger near and smaller far away", the proportion of the person in the picture naturally decreases. The third threshold can adapt to the characteristics of the proportion of the person in the medium panoramic category. If the proportion of the picture corresponding to the person category is greater than or equal to the third threshold and less than the second threshold, the person category is determined to be a medium panoramic category.

[0096] In order to fully accommodate these rich elements, the person's position relative to the lens must be farther away. Following the basic rule of "the image area is larger when it is near and smaller when it is far away", the proportion of the person in the picture naturally decreases. The third threshold can adapt to the characteristics of the proportion of the person in the medium panorama. As long as the proportion of the picture corresponding to the person category is increased to greater than or equal to the third threshold, it can be judged as a medium panorama category.

[0097] In step A4, the panorama category focuses on being able to fully display the full body shape of the person and the entire surrounding environment in which he is located. The person, as one of the core visual elements in the picture, contrasts with the surrounding environment, and together constructs a picture scene that is rich in content and comprehensive in information.

[0098] In the panoramic category, in order to fully present the characters and fully incorporate more environmental information, the distance between the characters and the lens is a little farther than that of the characters in the medium panoramic category. According to the first picture rule of "the image area is larger when it is near and smaller when it is far away", the proportion of the characters in the picture will be further reduced compared to the medium panoramic category, and will be within a specific range. The fourth threshold and the third threshold define this range. When the picture ratio corresponding to the character category falls in the interval greater than or equal to the fourth threshold and less than the third threshold, it just meets the size and proportion of the characters in the panoramic category. This means that at this time, the characters in the picture present a visual effect that can fully show the whole body and have an appropriate proportional relationship with the environment, so that the corresponding first scene category can be accurately determined as a panoramic category, providing an accurate and reasonable basic classification for subsequent further subdivision and precise judgment of scene.

[0099] In step A5, the distant view category emphasizes the grand and vast environment in which the characters are located. The characters play more of an embellishment role in the picture, used to reflect the scale, atmosphere and artistic conception of the environment.

[0100] In the distant view category, since the main focus is on the grand environment, the distance between the characters and the camera lens is very far. According to the rule that "the image area is larger near and smaller far", the characters occupy a very small proportion in the picture. Therefore, when the picture ratio corresponding to the character category is less than the fourth threshold, it clearly shows that the size of the characters in the picture is very small, which fully meets the typical feature of the distant view category that the characters occupy a small proportion in the picture. Based on this, the character category can be determined as the distant view category.

[0101] In summary, steps A1 to A5 set the first threshold, the second threshold, the third threshold and the fourth threshold that decrease in sequence, and determine the first scene category corresponding to the character category according to different picture ratio ranges. This can more accurately make a detailed preliminary distinction of scenes based on the actual proportion of the characters in the picture, in line with the rule that "the image area is larger when it is near and smaller when it is far away", laying a good foundation for the subsequent entire scene recognition process.

[0102] In an example, the first threshold may be 26%, the second threshold may be 11%, the third threshold may be 8%, and the fourth threshold may be 0.8%. It is understood that the embodiment of the present invention does not limit the specific values ​​of the first threshold, the second threshold, the third threshold, and the fourth threshold.

[0103] In step 103, the first scene category initially determined is corrected according to the posture key points corresponding to the above-mentioned character category, which can make up for the deficiency of scene recognition based only on the first picture rule, thereby improving the accuracy of scene recognition.

[0104] For example, in actual shooting scenes, people are often partially blocked. If the scene type is determined based on the first picture rule alone when there are many blocked parts, the picture ratio is prone to deviation. The reason is that large-area blocking seriously interferes with the presentation of the complete outline of the person in the picture, so that the obtained picture ratio cannot truly and accurately reflect the actual scene type.

[0105] The embodiment of the present invention can use posture recognition technology to perform posture recognition on the processed image to obtain the posture key points corresponding to each character category. Even if the character is partially blocked, the posture key point technology can reasonably predict the position of the posture key points corresponding to the blocked body parts based on the position combination characteristics of the visible body parts and the coherence of the human body structure. On this basis, these predicted posture key points are taken into consideration and used in the correction process of the first scene category, so as to effectively improve the accuracy of scene recognition and make the scene determination result more consistent with the actual shooting scene.

[0106] The posture key point technology can also reasonably predict the position of the posture key points corresponding to the occluded body parts based on the position combination characteristics of the visible body parts and the coherence of the human body structure, including the following aspects:

[0107] On the one hand, the prior knowledge of human body structure is used to preliminarily estimate the position of the occluded parts according to the fixed proportions of human body parts, the range of joint motion and symmetry characteristics;

[0108] On the other hand, by analyzing the spatial geometric relationship of body parts in the two-dimensional image, its direction can be further clarified with clues such as angle and direction;

[0109] On the other hand, with the help of probabilistic models, the probability distribution of body part combinations in different postures is used to assist in judgment. At the same time, deep learning methods are used to allow the probabilistic model to learn the characteristics of various occlusion situations and output prediction results after training with a large amount of data.

[0110] The above three aspects cooperate and work synergistically with each other to achieve accurate prediction of the positions of key points of the posture of the occluded parts.

[0111] The embodiment of the present invention does not limit the specific posture key point technology. For example, the posture key point technology may include: Dwpose (Dual-stage Distillation Whole-body Pose) estimation technology, etc.

[0112] In the embodiment of the present invention, the posture key points refer to those key points corresponding to the preset body parts, specifically including ankle key points, knee key points, hip key points, shoulder key points, nose key points and eye key points, etc. And it should be noted that each of the key points mentioned above may appear in pairs. For example, the ankle key point includes two key points corresponding to the left and right ankles, the knee key point also has two key points corresponding to the left and right knees, and the same is true for the hip and shoulder. Such paired key point settings help to more accurately depict the human posture and perform subsequent related analysis and judgment operations.

[0113] The embodiment of the present invention can provide the following technical solution for correcting the first scene category:

[0114] Technical solution 1,

[0115] In Technical Solution 1, the process of correcting the first scene category specifically includes: when both the ankle key points and the knee key points corresponding to the person category exist, if the first scene category is not a panoramic category or a distant view category, then the first scene category is updated to a panoramic category.

[0116] Technical solution 1 can make up for the deficiency that when scene recognition is performed only based on the first picture rule, the overall posture of the person and the actual scene cannot be accurately judged due to partial occlusion of the person's body or misleading picture composition.

[0117] In the case of only following the first picture rule, if the legs of the person in the picture are partially blocked, for example, there is a foreground object at the bottom of the picture blocking the ankle to the calf of the person, and the scene is calculated according to the conventional image area ratio, it is very likely that the picture that should be a panoramic view of the whole body posture will be mistakenly judged as a medium or close shot because of the inability to obtain complete leg information. In other words, because the lower part of the leg is missing, the overall area occupied by the person in the picture seems to be smaller, which meets the characteristics of a medium or close shot.

[0118] Technical Solution 1 starts with the key points of posture. As long as the ankle key points and the knee key points exist, it means that the key leg position information is complete and available. Even if the image to be processed has some interference and occlusion factors that affect the visual effect of the screen ratio, it can be clear that the posture of the lower body of the person is complete, and it should be in the category of panorama, which can show the whole body posture. If the first scene category does not match the panorama category, the first scene category is updated to the panorama category in time, correcting the deviation caused by relying solely on the screen area ratio to judge the scene due to occlusion or misleading composition.

[0119] Technical solution 2,

[0120] In Technical Solution 2, the process of correcting the first scene category specifically includes: when the ankle key point and the knee key point corresponding to the person category do not exist, and the hip key point corresponding to the person category exists, correcting the first scene category according to the positional relationship between the hip key point and the bounding box.

[0121] The bounding box refers to the circumscribed rectangular box of the person category in the image to be processed. The bounding box can be provided by the target detection technology. The bounding box can be represented by an array (or list) containing four values, which are the horizontal coordinate of the upper left corner vertex of the bounding box, the vertical coordinate of the upper left corner vertex, the horizontal coordinate of the lower right corner vertex, and the vertical coordinate of the lower right corner vertex.

[0122] It should be noted that object detection technology is a relatively coarse-grained recognition technology, and its bounding box may contain some background pixels that do not belong to the person category. Semantic segmentation technology is fine-grained, which accurately identifies and marks each pixel of the person category, achieving pixel-level understanding of the person category.

[0123] When a person's posture changes, such as leaning forward or bending, the area occupied by the person in the picture will change. In this case, judging the scene by area alone is prone to errors and cannot accurately reflect the actual scene. Moreover, in a complex background environment, the person and the background with similar colors and textures may be visually "fused", resulting in deviations in calculating the person's area, which in turn leads to errors in scene judgment.

[0124] When neither the ankle key point nor the knee key point corresponding to the person category exists, and the hip key point corresponding to the person category exists, the first scene category is corrected according to the positional relationship between the hip key point and the bounding box, which can improve the accuracy of scene recognition.

[0125] On the one hand, the hip key point reflects the second-picture rule that "the details of the character's posture are presented in different patterns under different shot sizes". This second-picture rule can accurately capture the characteristics of the character's posture and effectively correct misjudgments caused by the failure of conventional judgment factors.

[0126] On the other hand, the positional relationship between the hip key point and the bounding box reflects whether there is an occlusion problem with the hip key point. Based on this, we can quickly understand the situation where the integrity of the picture is affected, flexibly adjust the scene, and avoid the judgment deviation caused by occlusion. Specifically, when the hip key point is inside the bounding box, it can be basically determined that there is no occlusion interference on the hip, and the scene judgment can be carried out normally based on this; if it is outside the bounding box, it can be quickly known that there is occlusion on the hip, causing the character's posture to be incomplete, so as to adjust the scene in a timely and flexible manner, effectively avoiding the scene judgment deviation caused by occlusion factors.

[0127] In a specific implementation, whether the hip key point is inside the bounding box can be determined based on the size relationship between the coordinates of the hip key point and the coordinates of the bounding box.

[0128] Assuming that the coordinate system is a two-dimensional rectangular coordinate system with the upper left corner of the image as the origin, the horizontal rightward direction is the positive direction of the x-axis, and the vertical downward direction is the positive direction of the y-axis, then the y coordinate of the hip key point can be compared with the y coordinate of the bounding box. When the y coordinate of the hip key point is less than or equal to the y coordinate corresponding to the lower right corner of the bounding box, it can be determined that the hip key point is inside the bounding box. Conversely, when the y coordinate of the hip key point is greater than the y coordinate corresponding to the lower right corner of the bounding box, it can be determined that the hip key point is outside the bounding box.

[0129] In a specific implementation, when neither the ankle key point nor the knee key point corresponding to the person category exists, and the hip key point corresponding to the person category exists, the process of correcting the first scene category according to the positional relationship between the hip key point and the bounding box specifically includes:

[0130] Step B1: when neither the ankle key point nor the knee key point corresponding to the person category exists and the hip key point corresponding to the person category is located inside the bounding box, if the first scene category is not a medium panoramic category, then the first scene category is updated to a medium panoramic category; and / or

[0131] Step B2: When neither the ankle key point nor the knee key point corresponding to the person category exists and the hip key point corresponding to the person category is outside the bounding box, if the first scene category is not a medium scene category, the first scene category is updated to a medium scene category.

[0132] Among them, in step B1, when the ankle and knee key points corresponding to the character category do not exist and the hip key point is firmly inside the bounding box, it means that there is a high probability that there is no occlusion interference on the hip (corresponding to the case where the hip key point is not blocked). If the first scene category initially determined is not a medium panoramic category, since the complete presentation of the hip in the picture meets the requirements of the medium panoramic category for the completeness of the character subject, the first scene category is updated to a medium panoramic category to correct the scene deviation that may be caused by inaccurate early judgment factors.

[0133] It should be noted that if the first scene category is a medium panoramic category, the medium panoramic category may be kept unchanged.

[0134] In step B2, when the character's posture is presented as the ankle and knee key points are missing, and the hip key point is outside the bounding box, it means that the hip may be blocked, causing it to be outside the bounding box. If the original first scene category does not belong to the medium scene category, based on the picture state reflected by the current hip key point position, the first scene category is adjusted and updated to the medium scene category, so that the scene determination is more in line with the actual presentation of the character in the picture, and the overall scene recognition accuracy is improved.

[0135] In summary, the embodiment of the present invention innovatively combines the posture key points and the bounding box, which plays a key role. On the one hand, by accurately judging the positional relationship between the hip key points and the bounding box, the key condition of whether the hip is blocked or not can be quickly and accurately identified, which effectively avoids the misjudgment of the scene caused by not considering the occlusion factor, making the scene correction more targeted; on the other hand, in the complex case of missing ankle and knee key points, the first scene category is efficiently corrected according to the hip position relationship, whether it is accurately updating the non-medium panoramic category to the medium panoramic category (step B1), or reasonably adjusting the non-medium scene category to the medium scene category (step B2), which greatly improves the accuracy and adaptability of scene recognition.

[0136] Overall, this combination improves the scene judgment system, allowing it to stably output scene results that are more in line with the actual situation when faced with various types of character postures. It provides reliable technical support for many fields such as video processing and security monitoring image analysis that rely on accurate scene analysis, and effectively promotes related applications to move in a smarter and more accurate direction.

[0137] Technical solution 3,

[0138] In Technical Solution 3, the process of correcting the first scene category specifically includes: when the ankle key point, knee key point and hip key point corresponding to the person category do not exist, and the shoulder key point corresponding to the person category exists, correcting the first scene category according to the relationship between the first distance between the shoulder key point and the lower edge of the image to be processed and the semantic height of the person.

[0139] When the ankle, knee and hip key points corresponding to the person category do not exist, but the shoulder key point exists, Technical Solution 3 corrects the scene category based on the relationship between the first distance between the shoulder key point and the lower edge of the image to be processed and the semantic height of the person.

[0140] The character semantic height is based on the theoretical height range that the character should occupy when the character is fully presented in the picture and the semantics are clear. The character semantic height can be determined based on the absolute value of the difference between the maximum y coordinate and the minimum y coordinate of the pixel points in the character category. Alternatively, the character semantic height can be determined based on the height of the bounding box of the character category.

[0141] The shoulder key point and the lower edge of the image to be processed may be an absolute value of a difference between the y coordinate of the shoulder key point and the y coordinate of the lower edge of the image to be processed.

[0142] In a specific implementation, if the first distance is greater than or equal to the product of the human semantic height and the first ratio threshold, the scene category of the character category is corrected to a medium-close-up scene category. In this case, the extension of the character category in the vertical dimension of the picture meets the characteristic requirements of the medium-close-up scene category. The character's position in the picture is relatively high, and it occupies a large space vertically, which can show a certain range of body postures and surrounding environment details, so the scene category of the character category is accurately corrected to a medium-close-up scene category.

[0143] If the first distance is less than the product of the human semantic height and the first ratio threshold, the scene category of the character category is corrected to the close-up category. In this case, the main body of the character is more concentrated in the bottom area of ​​the picture, the vertical proportion is reduced, and the local details of the character, such as facial expressions or upper body movements, are highlighted, which is highly consistent with the characteristics of the close-up category that focuses on the key parts of the character. Therefore, correcting the scene category of the character category to the close-up category can avoid the scene misclassification caused by initial judgment errors or complex factors of the picture.

[0144] Those skilled in the art may determine the first ratio threshold according to actual application requirements. For example, the first ratio threshold may be a threshold such as 1 / 5.

[0145] Technical solution 4,

[0146] In technical solution 4, the process of correcting the first scene category specifically includes:

[0147] Step C1, when the ankle key point, knee key point, hip key point and shoulder key point corresponding to the character category do not exist, and the nose key point and eye key point corresponding to the character category exist, determine the horizontal distance and vertical distance between the eye key point and the nose key point, determine the horizontal length of the character and the semantic height of the character, and correct the first scene category according to the relationship between the horizontal distance and the horizontal length of the character, and the relationship between the vertical distance and the semantic height of the character; or

[0148] Step C2: if the ankle key point, knee key point, hip key point, shoulder key point and nose key point corresponding to the person category do not exist, and the eye key point corresponding to the person category exists, update the first scene category to the eye close-up category.

[0149] In step C1, the scene category is corrected by comparing the relationship between the horizontal distance between the eyes and the nose and the horizontal length of the character, and the relationship between the vertical distance and the semantic height of the character.

[0150] The horizontal distance can be understood as the distance between the eyes and the nose in the left-right direction of the image (usually the x-axis direction), while the vertical distance is the distance between the eyes and the nose in the up-down direction (usually the y-axis direction).

[0151] The horizontal length of a character refers to the length that a character should theoretically occupy in the horizontal direction. The horizontal length of a character can be determined based on the absolute value of the difference between the maximum x-coordinate and the minimum x-coordinate of the pixel points within the character category.

[0152] The relationship between the horizontal distance and the horizontal length of the character may be: a first ratio of the horizontal distance to the horizontal length of the character. The relationship between the vertical distance and the semantic height of the character may be: a second ratio of the vertical distance to the semantic height of the character.

[0153] The embodiment of the present invention can select the minimum value between the first ratio and the second ratio. If the minimum value is greater than the second ratio threshold, the character category is determined to be a face close-up category. Otherwise, if the minimum value is less than or equal to the second ratio threshold, the character category is determined to be an eye close-up category.

[0154] When the selected minimum value is greater than the second ratio threshold, it means that whether in the horizontal direction or the vertical direction, the relative distance between the eyes and the nose accounts for a sufficiently large proportion of the theoretical length or height of the corresponding dimension of the character. This means that the picture is highly focused on the character's face as a whole, and all key parts of the character's face are presented clearly and relatively enlarged, which meets the characteristics of the face close-up category (face close-up), so the character category is determined to be the face close-up category.

[0155] If the minimum value is less than or equal to the second ratio threshold, it indicates that in at least one of the horizontal and vertical dimensions, the relative spacing between the eyes and the nose is not prominent enough, and the overall image is more focused on the eyes, which is more consistent with the feature of the eye close-up category that focuses extremely on the eyes. Therefore, the character category is determined to be the eye close-up category.

[0156] Through this judgment mechanism based on proportional relationships and minimum value comparison, it is possible to more accurately correct the scene category based on the presentation of key facial parts in different character postures and complex scene scenes, improve the accuracy and reliability of scene recognition, and overcome the deviation problem that is easy to occur when judging scene by relying on a single factor.

[0157] In step C2, when the ankle key point, knee key point, hip key point, shoulder key point and nose key point corresponding to the person category do not exist, and the eye key point corresponding to the person category exists, it can be inferred that the picture is mainly focused on the eyes. Because in normal shooting or image processing situations, if only the eyes are retained or identifiable as key information points, it is likely because the focus of the picture is the eyes of the person. This focus state meets the characteristics of the eye close-up category.

[0158] Technical solution 5,

[0159] In technical solution 5, the process of correcting the first scene category specifically includes:

[0160] Step D1, judging whether the image to be processed has screen tilt according to the posture key points corresponding to the character category;

[0161] Step D2: If the image to be processed is tilted, the first scene category is updated from long shot to full shot, or the first scene category is updated from full shot to medium full shot, or the first scene category is updated from medium full shot to medium shot, or the first scene category is updated from medium shot to medium close shot, or the first scene category is updated from medium close shot to close shot, or the first scene category is updated from close shot to face close-up, or the first scene category is updated from face close-up to eye close-up.

[0162] When the picture is tilted, the visual presentation effect of the character in the picture will change. From the distant view category to the eye close-up category, the proportion of characters in different scenes and the degree of detail presentation in the picture gradually increase. The tilt of the picture will change the effective visible part of the character in the picture, and it seems that the proportion of the character in the picture has increased. For example, in the distant view, the tilt of the picture may cause a part of the main body of the character (such as the head and upper body) to visually occupy a larger proportion that would not be occupied in the distant view, which is closer to the panoramic character proportion effect. Therefore, in the embodiment of the present invention, when the image to be processed is tilted, the scene category is updated and zoomed in according to specific rules. Not only can the details of the character that were originally blurred and difficult to distinguish due to the tilt gradually become clear as the scene is zoomed in, and the character information can be fully and accurately transmitted to avoid misunderstandings; but it can also make up for the visual imbalance caused by the tilt of the picture, refocus the viewer's sight, and improve the viewing comfort and attractiveness.

[0163] The process of judging whether the image to be processed has a tilted image specifically includes:

[0164] First, obtain a pair of posture key points (such as the left and right eyes), determine the vector of the line connecting the two coordinate points according to the two coordinate points of the pair of posture key points, and normalize them to obtain a first vector;

[0165] Secondly, the dot product of the first vector and the y-axis unit vector of the screen is calculated to obtain the dot product result;

[0166] Furthermore, if the dot product result is greater than the eleventh threshold, it is considered that the picture is tilted. Otherwise, if the dot product result is less than or equal to the eleventh threshold, the picture is not tilted.

[0167] Among them, the dot product result reflects the angle relationship between the line connecting the selected pairs of posture key points and the vertical direction of the picture.

[0168] If the dot product result is greater than the eleventh threshold, it means that the line connecting the selected pairs of posture key points has a more obvious deviation from the vertical direction of the picture (y-axis direction), and the degree of deviation exceeds the normal range, so it is judged that the picture is tilted; conversely, if the dot product result is less than or equal to the eleventh threshold, it means that the deviation of the line connecting the paired posture key points from the vertical direction of the picture is within an acceptable range, and it is considered that the picture is not tilted, and the posture of the overall image is in a relatively normal state.

[0169] The process of correcting the first scene category is described in detail above through Technical Solutions 1 to 5. It can be understood that those skilled in the art can adopt any one or a combination of Technical Solutions 1 to 5 according to actual application requirements.

[0170] It should be noted that the image to be processed may contain one or more person categories, wherein the first scene category of some or all person categories in the image to be processed may be corrected.

[0171] In the actual process of scene correction for the image to be processed, the determining factor lies in the fit between the posture key points of each character category in the image and the correction conditions. Different character categories have their own corresponding posture key point distribution in the image to be processed, and Technical Solutions 1 to 5 set clear and specific correction conditions respectively. These correction conditions are often defined around the existence or non-existence of various posture key points such as ankles, knees, hips, shoulders, noses, eyes, etc. and the positional relationship between them.

[0172] When the state presented by the posture key points of a certain character category just meets the correction conditions specified in the above-mentioned specific technical solution, this means that for this character category, the basis and premise for correcting its first scene category according to the corresponding technical solution are met. Based on this, it is possible to carry out specific correction operations on the first scene category of these character categories that meet the conditions, and use corresponding rules and steps to adjust their scene classification to make it more in line with the actual presentation state of the character category in the image to be processed, thereby improving the accuracy of scene determination.

[0173] The embodiment of the present invention cleverly utilizes the matching relationship between posture key points and correction conditions, and through rigorously judging the compliance of each character category in this regard, specifically screens out those character categories that can be corrected for scene scale. This method is highly flexible and practical, and can accurately determine which character categories need to be corrected for scene scale based on the diverse characteristics of different images and the differentiated processing requirements in specific application scenarios.

[0174] Therefore, in the embodiment of the present invention, the scene recognition result of an image to be processed may include: a first scene category corresponding to the person category, and / or a second scene category corresponding to the person category.

[0175] In the embodiment of the present invention, the scene recognition result of the image to be processed can provide data support for the downstream image annotation task. In the process of executing the downstream image annotation task, the annotation information of the image to be processed can be automatically generated according to the scene recognition result, which can not only improve the efficiency of data annotation, but also improve the accuracy of data annotation.

[0176] In an embodiment of the present invention, image data annotation may refer to the process of marking and classifying human objects in an image, with the aim of enabling a machine learning model to recognize and understand the scene characteristics of human objects in the image.

[0177] The scene recognition result obtained in the embodiment of the present invention may be a pixel-level result, which may specifically include: pixel point positions and scene category information. The image annotation task may automatically generate annotation information of the image to be processed based on the pixel point positions and scene category information. Specifically, the pixel point positions and scene category information may be converted into annotation information in a preset format.

[0178] The above-mentioned preset format may be a format that can be recognized by the machine learning model. The above-mentioned preset format may include: extensible markup language format, JavaScript object notation format, etc. It can be understood that the embodiment of the present invention does not limit the specific preset format.

[0179] The embodiment of the present invention can generate a scene terminology dataset based on a plurality of images to be processed and their corresponding scene recognition results. Specifically, these images to be processed and their corresponding scene recognition results are integrated and summarized to obtain a complete scene terminology dataset.

[0180] The above-mentioned scene terminology dataset can meet the data volume demand in the field of AIGC (Artificial Intelligence-Generated Content).

[0181] In the field of AIGC, the scene terminology dataset is the training cornerstone of the generative model. For the generative model that generates images, videos and other content, this dataset built based on the scene recognition results provides the generative model with rich visual information. The generative model can understand the visual patterns contained in different scene categories by learning the features of the characters, postures and positional relationships of the characters in different scene categories in the dataset. For example, the generative model can learn that the facial details of the characters in the close-up are rich and they occupy a large proportion of the picture, while the characters in the distant view are relatively small and have more information about the surrounding environment. This learning process is similar to how humans understand the concept of scene by observing a large number of images, so that the generative model can reasonably arrange the layout of the characters in the picture according to the learned scene knowledge when generating content in the future.

[0182] Furthermore, since the embodiment of the present invention selects images to be processed whose resolution and clarity meet the requirements, on this basis, the embodiment of the present invention can provide a high-quality clear data set for the AIGC field.

[0183] In addition, the scene terminology dataset obtained in the embodiment of the present invention can not only be used for training the generation model, but also can be used for creating a film and television database, which is convenient for users to search for corresponding dramas in a targeted manner and is convenient for later shooting reference and advertising insertion.

[0184] The film and television database aims to comprehensively collect data related to various film and television works. In addition to the common basic information of the film (such as the title, director, starring, etc.) and the plot introduction, the scene information is also an important part of describing the content characteristics of film and television works. Using this scene terminology dataset, the scene conditions of each scene and lens in different film and television works can be recorded in detail, thus constructing a film and television database that covers rich scene details, making the content of the film and television database more three-dimensional and comprehensive.

[0185] When users want to find dramas with specific scene types, this film and television database built based on the scene terminology dataset can play a big role. For example, a student majoring in film and television wants to study the application techniques of close-up shots (such as face close-up categories, eye close-up categories, etc.) in film and television works. In the past, he may only be able to search by manually watching and screening each episode, which is very inefficient. However, with a film and television database containing scene information, you can directly use the search function to enter the scene keywords you want to find (such as "close-up shot" and "close-up picture", etc.), and the film and television database can quickly filter out dramas and specific scene clips that meet the requirements, so that users can accurately locate the content they are interested in, greatly improving the efficiency of finding film and television materials related to specific scenes, and meeting the needs of different users for targeted searches based on scene types.

[0186] In summary, the scene recognition method of the embodiment of the present invention is automatically operated by a computer program. Specifically, the computer program first determines the person category corresponding to the pixel point in the image to be processed, then determines the first scene category based on the objective screen ratio of the person category in the image, and then corrects the first scene category using the posture key points corresponding to the person category.

[0187] Since the method of the embodiment of the present invention can save tedious and time-consuming manual analysis work, it can save the labor cost of scene recognition. In addition, the method makes use of the powerful computing power and fast data processing characteristics of computers, so the embodiment of the present invention can improve the efficiency of scene recognition.

[0188] Furthermore, the embodiment of the present invention first determines the first scene category by using the first picture rule of "the image area is larger when near and smaller when far away"; and then corrects the first scene category by using the second picture rule of "the presentation rules of the details of the person's posture are different under different scenes". The combination of the first picture rule and the second picture rule can comprehensively determine the scene category corresponding to the person category from two dimensions: macroscopic (screen ratio) and microscopic (posture details). Therefore, the embodiment of the present invention can improve the accuracy and reliability of the scene recognition results.

[0189] Reference Figure 2 , shows a flowchart of a scene recognition method according to an embodiment of the present invention, the method may specifically include the following steps:

[0190] Step 201: Acquire an image to be processed whose resolution and clarity meet set conditions from an image set;

[0191] Step 202, determining the person category corresponding to the pixel point in the image to be processed;

[0192] Step 203: determining the first scene category corresponding to the above character category according to the picture ratio corresponding to the above character category;

[0193] Step 204, determine whether both the ankle key point and the knee key point corresponding to the character category exist, if so, execute step 205, otherwise execute step 206;

[0194] Step 205: determine whether the first scene category corresponding to the person category is a panoramic category or a distant view category, and if so, update the first scene category to a panoramic category;

[0195] It can be understood that if the first scene category is not a panoramic category or a distant scene category, the first scene category can be kept unchanged.

[0196] Step 206, determine whether the hip key point corresponding to the character category exists, if yes, execute step 207, otherwise execute step 210;

[0197] Step 207, determining whether the hip key point corresponding to the character category is located inside the bounding box, if so, executing step 208, otherwise executing step 209;

[0198] Step 208: If the first scene category is not a medium panoramic category, then update the first scene category to a medium panoramic category;

[0199] It can be understood that if the first scene category is a medium panoramic category, the medium panoramic category can be kept unchanged.

[0200] Step 209: If the first scene category is not a medium scene category, then update the first scene category to a medium scene category;

[0201] It can be understood that if the first scene category is a mid-shot category, the mid-shot category can be kept unchanged.

[0202] Step 210, determine whether the shoulder key point corresponding to the character category exists, if yes, execute step 211, otherwise execute step 212;

[0203] Step 211, correcting the first scene category according to the relationship between the first distance between the shoulder key point and the lower edge of the image to be processed and the semantic height of the character;

[0204] Step 212, determine whether the nose key point corresponding to the character category exists, if yes, execute step 213, otherwise execute step 214;

[0205] Step 213, correcting the first scene category according to the relationship between the horizontal distance and the horizontal length of the character, and the relationship between the vertical distance and the semantic height of the character;

[0206] Step 214: Update the first scene category to the eye close-up category;

[0207] After step 205, the above method may further include:

[0208] Step 215: If the image to be processed is tilted, zoom in on the first scene category or the second scene category.

[0209] The above-mentioned zoom-in processing specifically includes: updating the first scene category from long shot to full shot, or updating the first scene category from full shot to medium full shot, or updating the first scene category from medium full shot to medium shot, or updating the first scene category from medium shot to medium close shot, or updating the first scene category from medium close shot to close shot, or updating the first scene category from close shot category to face close-up category, or updating the first scene category from face close-up category to eye close-up category.

[0210] It should be noted that if the image to be processed does not have a screen tilt, the first scene category or the second scene category may be kept unchanged.

[0211] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0212] Reference Figure 3 , shows a schematic diagram of the structure of a scene recognition device according to an embodiment of the present invention. The scene recognition device may specifically include the following modules: a person category determination module 301, an initial scene determination module 302 and a scene correction module 303.

[0213] The character category determination module 301 is used to determine the character category corresponding to the pixel point in the image to be processed;

[0214] An initial scene determination module 302 is used to determine a first scene category corresponding to the person category according to the picture ratio corresponding to the person category; the picture ratio is the ratio of the image area of ​​the person category to the total image area of ​​the image to be processed;

[0215] The scene correction module 303 is used to correct the first scene category according to the posture key points corresponding to the character category to obtain a second scene category corresponding to the character category.

[0216] Optionally, the scene correction module includes:

[0217] The first scene correction module is used to update the first scene category to a panoramic category if both the ankle key points and the knee key points corresponding to the person category exist and if the first scene category is not a panoramic category or a distant view category.

[0218] Optionally, the scene correction module includes:

[0219] The second scene correction module is used to correct the first scene category according to the positional relationship between the hip key point and the bounding box when the ankle key point and the knee key point corresponding to the character category do not exist and the hip key point corresponding to the character category exists.

[0220] Optionally, the second scene correction module includes:

[0221] A first updating module is used for updating the first scene category to a medium panoramic category if the first scene category is not a medium panoramic category when neither the ankle key point nor the knee key point corresponding to the person category exists and the hip key point corresponding to the person category is located inside a bounding box; the bounding box is an external rectangular box corresponding to the person category in the image to be processed; and / or

[0222] The second updating module is used to update the first shot category to the medium shot category if the first shot category is not the medium shot category, when neither the ankle key point nor the knee key point corresponding to the character category exists and the hip key point corresponding to the character category is outside the bounding box.

[0223] Optionally, the scene correction module includes:

[0224] The third scene correction module is used to correct the first scene category according to the relationship between the first distance between the shoulder key point and the lower edge of the image to be processed and the semantic height of the character when the ankle key point, knee key point and hip key point corresponding to the character category do not exist and the shoulder key point corresponding to the character category exists.

[0225] Optionally, the scene correction module includes:

[0226] A fourth scene correction module is used to determine the horizontal distance and vertical distance between the eye key point and the nose key point, determine the horizontal length of the character and the semantic height of the character, and correct the first scene category according to the relationship between the horizontal distance and the horizontal length of the character, and the relationship between the vertical distance and the semantic height of the character, when the ankle key point, knee key point, hip key point and shoulder key point corresponding to the character category do not exist, and the nose key point and eye key point corresponding to the character category both exist; or

[0227] The fifth scene correction module is used to update the first scene category to the eye close-up category when the ankle key point, knee key point, hip key point, shoulder key point and nose key point corresponding to the person category do not exist, and the eye key point corresponding to the person category exists.

[0228] Optionally, the scene correction module includes:

[0229] A tilt judgment module, used for judging whether the image to be processed has a tilt according to the posture key points corresponding to the character category;

[0230] The sixth scene correction module is used to update the first scene category from distant view to panoramic view, or the first scene category from panoramic view to medium panoramic view, or the first scene category from medium panoramic view to medium scene, or the first scene category from medium scene to medium close scene, or the first scene category from medium close scene to close scene, or the first scene category from close scene to face close-up category, or the first scene category from face close-up category to eye close-up category, if there is a picture tilt in the image to be processed.

[0231] Optionally, the preliminary scene determination module includes:

[0232] A first preliminary scene determination module is configured to determine the first scene category corresponding to the character category as a close-up scene category when the picture ratio corresponding to the character category is greater than or equal to a first threshold; or

[0233] A second preliminary scene determination module is used to determine the first scene category corresponding to the person category as a medium or close scene category when the picture ratio corresponding to the person category is greater than or equal to the second threshold and less than the first threshold; or

[0234] A third preliminary scene determination module is configured to determine the first scene category corresponding to the person category as a medium panoramic category when the picture ratio corresponding to the person category is greater than or equal to the third threshold and less than the second threshold; or

[0235] a fourth preliminary scene determination module, configured to determine the first scene category corresponding to the person category as a panoramic category when the picture ratio corresponding to the person category is greater than or equal to the fourth threshold and less than the third threshold; or

[0236] a fifth preliminary scene determination module, configured to determine the first scene category corresponding to the person category as a distant scene category when the picture ratio corresponding to the person category is less than a fourth threshold;

[0237] Among them, the first threshold is greater than the second threshold, the second threshold is greater than the third threshold, and the third threshold is greater than the fourth threshold.

[0238] An embodiment of the present invention further provides an electronic device that can implement the functions of the aforementioned storage node or scheduling node or ledger node or management node.

[0239] like Figure 4 As shown, the electronic device may include a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other via the communication bus 1004.

[0240] Memory 1003, used for storing computer programs;

[0241] The processor 1001 is used to implement the following steps when executing the program stored in the memory 1003:

[0242] Storing the storage object;

[0243] Sending the storage object to a coupled second storage node, so that the second storage node stores and / or forwards the storage object;

[0244] After the storage object is successfully stored, the object information of the storage object is sent to the ledger node, so that the ledger node saves the ledger information of the first storage node; the ledger information includes: the object information stored in the corresponding storage node.

[0245] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0246] The communication interface is used for communication between the above electronic device and other devices.

[0247] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one scene recognition device located away from the aforementioned processor.

[0248] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0249] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the scene recognition method described in any one of the above embodiments.

[0250] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the scene recognition method described in any one of the above embodiments.

[0251] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc. Examples of optical media may include DVD (Digital Video Disc) and the like.

[0252] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0253] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0254] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A scene recognition method, characterized in that: The method comprises: Determine the person category corresponding to the pixel point in the image to be processed; Determine the first scene category corresponding to the person category according to the picture ratio corresponding to the person category; the picture ratio is the ratio of the image area of ​​the person category to the total image area of ​​the image to be processed; According to the posture key points corresponding to the character category, the first scene category is corrected to obtain a second scene category corresponding to the character category.

2. The method according to claim 1, characterized in that The correcting the first scene category according to the posture key points corresponding to the character category includes: In the case that both the ankle key points and the knee key points corresponding to the person category exist, if the first scene category is not a panoramic category or a distant view category, the first scene category is updated to a panoramic category.

3. The method according to claim 1, characterized in that The correcting the first scene category according to the posture key points corresponding to the character category includes: When neither the ankle key point nor the knee key point corresponding to the person category exists, and the hip key point corresponding to the person category exists, the first scene category is corrected according to the positional relationship between the hip key point and the bounding box.

4. The method according to claim 3, characterized in that The method of correcting the first scene category according to the positional relationship between the hip key point and the bounding box, when neither the ankle key point nor the knee key point corresponding to the person category exists, includes: In the case that neither the ankle key point nor the knee key point corresponding to the person category exists, and the hip key point corresponding to the person category is located inside the bounding box, if the first scene category is not a medium panoramic category, then the first scene category is updated to a medium panoramic category; the bounding box is an external rectangular box corresponding to the person category in the image to be processed; and / or When neither the ankle key point nor the knee key point corresponding to the person category exists, and the hip key point corresponding to the person category is outside the bounding box, if the first scene category is not a medium scene category, the first scene category is updated to a medium scene category.

5. The method according to claim 1, characterized in that The correcting the first scene category according to the posture key points corresponding to the character category includes: When the ankle key point, knee key point and hip key point corresponding to the person category do not exist, and the shoulder key point corresponding to the person category exists, the first scene category is corrected according to the relationship between the first distance between the shoulder key point and the lower edge of the image to be processed and the semantic height of the person.

6. The method according to claim 1, characterized in that The correcting the first scene category according to the posture key points corresponding to the character category includes: When the ankle key point, knee key point, hip key point and shoulder key point corresponding to the character category do not exist, and the nose key point and eye key point corresponding to the character category exist, determine the horizontal distance and vertical distance between the eye key point and the nose key point, determine the horizontal length of the character and the semantic height of the character, and correct the first scene category according to the relationship between the horizontal distance and the horizontal length of the character, and the relationship between the vertical distance and the semantic height of the character; or When the ankle key point, knee key point, hip key point, shoulder key point and nose key point corresponding to the person category do not exist, and the eye key point corresponding to the person category exists, the first scene category is updated to the eye close-up category.

7. The method according to claim 1, characterized in that The correcting the first scene category according to the posture key points corresponding to the character category includes: Judging whether the image to be processed has a screen tilt according to the posture key points corresponding to the character category; If the image to be processed is tilted, the first scene category is updated from long shot to full shot, or the first scene category is updated from full shot to medium full shot, or the first scene category is updated from medium full shot to medium shot, or the first scene category is updated from medium shot to medium close shot, or the first scene category is updated from medium close shot to close shot, or the first scene category is updated from close shot to face close-up category, or the first scene category is updated from face close-up category to eye close-up category.

8. The method according to any one of claims 1 to 7, characterized in that: The step of determining the first scene category corresponding to the character category according to the picture ratio corresponding to the character category includes: According to the picture ratio corresponding to the character category, a search is performed in the correspondence between the picture ratio range and the scene category to obtain the first scene category corresponding to the character category.

9. A scene recognition device, characterized in that: The device comprises: A character category determination module is used to determine the character category corresponding to the pixel point in the image to be processed; An initial scene determination module is used to determine a first scene category corresponding to the character category according to the picture ratio corresponding to the character category; the picture ratio is the ratio of the image area of ​​the character category to the total image area of ​​the image to be processed; The scene correction module is used to correct the first scene category according to the posture key points corresponding to the character category to obtain a second scene category corresponding to the character category.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 8 when executing a program stored in a memory.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method described in claims 1-8 is implemented.

Images