Image recognition method, device, equipment and storage medium

Abstract

The embodiments of the present application provide an image recognition method, apparatus, device and storage medium, which relate to the field of artificial intelligence technology. The method includes: detecting at least one target object in the image to be recognized, obtaining recognition frames corresponding to at least one target object, and determining the partial incomplete recognition results of each recognition frame obtained. Based on the area of each recognition frame, the target analysis object is determined from at least one target object. Based on the partial incomplete recognition results of each recognition frame corresponding to the target analysis object, the overall recognition result of the target analysis object is determined. Based on the overall recognition result of the target analysis object, the image recognition result of the image to be recognized is determined. The completeness of the target analysis object is determined by the characteristics of the target analysis object in multiple aspects, thereby improving the accuracy of identifying the completeness of the target analysis object, and further improving the accuracy and recall rate of detecting whether there is an object incompleteness problem in the image to be recognized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more particularly to an image recognition method, apparatus, device, and storage medium. Background Art

[0002] With the popularity and growth of information streams and short videos, a large number of self-media outlets and ordinary users have begun to create and publish multimedia content such as images and videos. However, some multimedia content is not original but rather generated through secondary processing. During this secondary processing, due to the different image sizes and ratios supported by different platforms, users often crop the original images. This results in incomplete images and human figures in some videos and images due to cropping, which in turn affects the viewing experience.

[0003] Related technologies detect key points of human bodies in images to determine whether an image contains incomplete figures. However, in real-world applications, images are rich in content, and many of these features can lead to erroneous key point detection results, resulting in low accuracy in identifying incomplete figures. Summary of the Invention

[0004] Embodiments of the present application provide an image recognition method, apparatus, device, and storage medium for improving recognition accuracy when identifying whether an image has an incomplete object problem.

[0005] In one aspect, an embodiment of the present application provides an image recognition method, the method comprising:

[0006] Detecting at least one target object in the image to be recognized, obtaining recognition frames corresponding to the at least one target object, and determining a partial incomplete recognition result of each recognition frame;

[0007] determining a target analysis object from the at least one target object based on the areas of the respective identification frames;

[0008] Determining an overall recognition result of the target analysis object based on the partial recognition results of each recognition frame corresponding to the target analysis object;

[0009] An image recognition result of the image to be recognized is determined based on the overall recognition result of the target analysis object.

[0010] In one aspect, an embodiment of the present application provides an image recognition device, comprising:

[0011] a detection module, configured to detect at least one target object in the image to be identified, obtain identification frames corresponding to the at least one target object, and determine a partial incomplete identification result of each identification frame;

[0012] A selection module, configured to determine a target analysis object from the at least one target object based on the areas of the respective identification frames;

[0013] A matching model is used to determine an overall recognition result of the target analysis object based on the partial recognition results of each recognition frame corresponding to the target analysis object;

[0014] The discrimination module is used to determine the image recognition result of the image to be recognized based on the overall recognition result of the target analysis object.

[0015] Optionally, the detection module is specifically configured to:

[0016] Using a trained target detection model, at least one target object in the image to be identified is detected to obtain at least one identification frame, and a partial incomplete identification result of the at least one identification frame is determined, wherein the sample image used for training the target detection model is obtained by randomly cropping an image from which the target object has been segmented;

[0017] Based on the positional relationship between the at least one recognition frame, the at least one recognition frame is grouped to obtain recognition frames corresponding to the at least one target object.

[0018] Optionally, the at least one recognition frame includes at least one target object recognition frame and at least one sub-target object recognition frame, the sub-target object being a partial area of the target object, and the partial incomplete recognition result of the at least one recognition frame includes a target object recognition result and a sub-target object recognition result, wherein the target object recognition result is used to indicate whether the target object within the target object recognition frame is complete, and the sub-target object recognition result is used to indicate whether the sub-target object within the sub-target object recognition frame is complete;

[0019] The detection module is specifically used for:

[0020] At least one target object in the image to be identified is detected using the trained target detection model to obtain at least one target object identification frame and at least one sub-target object identification frame, and a target object recognition result of the at least one target object identification frame and a sub-target object recognition result of the at least one sub-target object identification frame are determined.

[0021] Optionally, the target object recognition frame is a human body recognition frame, and the sub-target object recognition frame includes at least one of the following: a face recognition frame and a head recognition frame.

[0022] Optionally, the detection module is further configured to:

[0023] The at least one identification frame is grouped based on a positional relationship between the at least one identification frame. Before obtaining identification frames corresponding to the at least one target object, a screening operation is iteratively performed on a first screening object until the first screening object satisfies a first preset condition, wherein the initial first screening object is the at least one target object identification frame, and each iterative process includes the following screening operations:

[0024] Obtaining a first analysis reference recognition frame with the highest probability of a target object recognition result from the first screening objects;

[0025] Determine a first intersection-over-union ratio between the first analysis reference recognition frame and each of the other target object recognition frames in the first screening object;

[0026] Remove the target object recognition frames whose first intersection-over-union ratio is greater than a first threshold from the other target object recognition frames;

[0027] The first screening object is updated to the target object recognition frame retained in the other target object recognition frames.

[0028] Optionally, the detection module is further configured to:

[0029] Iteratively perform a screening operation on the second screening object until the second screening object satisfies a second preset condition, wherein the initial second screening object is the at least one sub-target object identification frame, and each iterative process includes the following screening operations:

[0030] Obtaining a second analysis reference recognition frame with the highest probability of sub-target object recognition result from the second screening objects;

[0031] Determine a second intersection-over-union ratio between the second analysis reference recognition frame and each of the other sub-target object recognition frames in the second screening object;

[0032] Remove the sub-target object identification frames whose second intersection-over-union ratio is greater than a second threshold from the other sub-target object identification frames;

[0033] The second screening object is updated to the sub-target object identification frame retained in each of the other sub-target object identification frames.

[0034] Optionally, the selection module is specifically configured to:

[0035] For each identification frame corresponding to each target object in the at least one target object, perform the following operations:

[0036] For each recognition frame corresponding to a target object, obtaining a candidate recognition frame with the largest area from among the recognition frames corresponding to the target object;

[0037] From the obtained candidate recognition frames, select the candidate recognition frame with the largest area as the target recognition frame;

[0038] The target object corresponding to the target identification frame in at least one target object is used as a target analysis object.

[0039] Optionally, the matching module is specifically configured to:

[0040] If the part incomplete recognition results of each recognition frame corresponding to the target analysis object all indicate that the parts are not incomplete, then the overall recognition result of the target analysis object is determined to be a complete object; otherwise, the overall recognition result of the target analysis object is determined to be an incomplete object.

[0041] Optionally, the discrimination module is specifically configured to:

[0042] If the overall recognition result of the target analysis object is that the object is complete, determining that the image recognition result of the image to be recognized is a complete image;

[0043] If the overall recognition result of the target analysis object is that the object is incomplete, then it is determined that the image recognition result of the image to be recognized is an incomplete image.

[0044] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned image recognition method when executing the program.

[0045] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned image recognition method.

[0046] In an embodiment of the present application, based on the area of the identification frame corresponding to at least one target object, the target analysis object is determined from at least one target object, and then based on the overall recognition result of the target analysis object, the image recognition result of the image to be identified is directly determined, without the need to identify the overall recognition result of each target object in the image, thereby improving the accuracy and efficiency of identifying whether there is an object incomplete problem in the image. Secondly, based on the partial incomplete recognition results of each identification frame corresponding to the target analysis object, the overall recognition result of the target analysis object is determined, that is, based on the characteristics of the target analysis object in multiple aspects, the completeness of the target analysis object is determined, thereby improving the accuracy of identifying the completeness of the target analysis object, and further improving the accuracy and recall rate of detecting whether there is an object incomplete problem in the image to be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A schematic diagram of a process for image, text or video review provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a cover image selection process provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a system architecture provided in an embodiment of the present application;

[0051] Figure 4 A flowchart of an image recognition method provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of an identification frame provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of an initial image provided in an embodiment of the present application;

[0054] Figure 7a A schematic diagram of a sample image provided in an embodiment of the present application;

[0055] Figure 7b A schematic diagram of a sample image provided in an embodiment of the present application;

[0056] Figure 8a A schematic diagram of a human body recognition frame provided in an embodiment of the present application;

[0057] Figure 8b A schematic diagram of a facial recognition frame provided in an embodiment of the present application;

[0058] Figure 9 A schematic diagram of a human body recognition frame, a head recognition frame, and a face recognition frame provided in an embodiment of the present application;

[0059] Figure 10 A schematic diagram of a human body recognition frame, a head recognition frame, and a face recognition frame provided in an embodiment of the present application;

[0060] Figure 11 A schematic diagram of a human body recognition frame provided in an embodiment of the present application;

[0061] Figure 12 A schematic diagram of a facial recognition frame provided in an embodiment of the present application;

[0062] Figure 13 A schematic diagram of a human body recognition frame, a head recognition frame, and a face recognition frame provided in an embodiment of the present application;

[0063] Figure 14 A flowchart of an image recognition method provided in an embodiment of the present application;

[0064] Figure 15 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0065] Figure 16 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and beneficial effects of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0067] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0068] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0069] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0070] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. For example, in the embodiment of the present application, computer vision technology is used to identify whether the target object in the image is complete.

[0071] The following is an introduction to the design concept of the embodiments of the present application.

[0072] With the popularity and growth of information streams and short videos, a large number of self-media outlets and ordinary users have begun to create and publish multimedia content such as images and videos. However, some multimedia content is not original but rather generated through secondary processing. During this secondary processing, due to the different image sizes and ratios supported by different platforms, users often crop the original images. This results in incomplete images and human figures in some videos and images due to cropping, which in turn affects the viewing experience.

[0073] Related technologies detect key points of human figures in an image to determine if an image contains incomplete figures. However, in real-world applications, images are often rich in content, and this often leads to errors in key point detection, resulting in low accuracy in identifying incomplete figures. Furthermore, when an image includes multiple figures, including both complete and incomplete figures, key point detection methods are even less accurate in identifying incomplete figures.

[0074] Our analysis found that when users view an image or video, if the main character's body is intact while the bodies of other secondary characters are incomplete, it does not affect the user's viewing experience. Furthermore, when detecting the integrity of a human body, examining it from multiple perspectives will help improve its accuracy.

[0075] In view of this, an embodiment of the present application provides an image recognition method, comprising: detecting at least one target object in an image to be recognized, obtaining identification frames corresponding to at least one target object, and determining partial incomplete recognition results for each identification frame. Then, based on the area of each identification frame, a target analysis object is determined from the at least one target object. Based on the partial incomplete recognition results for each identification frame corresponding to the target analysis object, an overall recognition result for the target analysis object is determined. Finally, based on the overall recognition result for the target analysis object, an image recognition result for the image to be recognized is determined.

[0076] In an embodiment of the present application, based on the area of the identification frame corresponding to at least one target object, the target analysis object is determined from at least one target object, and then based on the overall recognition result of the target analysis object, the image recognition result of the image to be identified is directly determined, without the need to identify the overall recognition result of each target object in the image, thereby improving the accuracy and efficiency of identifying whether there is an object incomplete problem in the image. Secondly, based on the partial incomplete recognition results of each identification frame corresponding to the target analysis object, the overall recognition result of the target analysis object is determined, that is, based on the characteristics of the target analysis object in multiple aspects, the completeness of the target analysis object is determined, thereby improving the accuracy of identifying the completeness of the target analysis object, and further improving the accuracy and recall rate of detecting whether there is an object incomplete problem in the image to be identified.

[0077] After introducing the design concepts of the embodiments of this application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of this application and are not limiting. In specific implementations, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0078] Application scenario 1: Image, text or video review scenario.

[0079] See Figure 1 When a user uploads or publishes text, image, or video content through a multimedia application, the backend uses the image recognition method in the embodiment of the present application to identify the completeness of the human body in the text, image, or video content. If the human body is incomplete in the text, image, or video content, the text, image, or video content will be blocked or downgraded. Otherwise, the text, image, or video content will be saved to the backend content recommendation pool.

[0080] Application scenario 2: Cover image selection scenario.

[0081] See Figure 2In information flow and short video products, in order to attract users to read, it is necessary to select a suitable and beautiful cover image from the text or video content. Specifically, after the user uploads or publishes text or video content through a multimedia application, multiple candidate cover images are obtained from the text or video content, and then the image recognition method in the embodiment of the present application is used to identify the multiple candidate cover images to determine the degree of completeness of the human body in each candidate cover image. If the candidate cover image has an incomplete human body, the candidate cover image is removed, otherwise the next step of the cover image selection process is entered, and the candidate cover image is handed over to the cover image processing module for processing.

[0082] Application scenario three: driver and passenger identity recognition scenario.

[0083] The vehicle-mounted terminal obtains an image of the driver or passenger captured by the camera, and then uses the image recognition method of the embodiment of the present application to identify the driver or passenger image and determine the completeness of the face in the driver or passenger image. If the face in the driver or passenger image is incomplete, the terminal notifies the user that the identity verification has failed and the vehicle cannot be started. Otherwise, the terminal determines the similarity between the facial features in the driver or passenger image and the previously uploaded facial features of the driver. If the similarity is greater than a preset threshold, the terminal notifies the user that the identity verification has passed and the vehicle can be started. Otherwise, the terminal notifies the user that the identity verification has failed and the vehicle cannot be started.

[0084] It should be noted that the image recognition method in the embodiments of the present application is not only applicable to the above-mentioned application scenarios, but can also be used in any other scenarios where the target object in the recognition image is incomplete. This application does not make any specific limitations on this.

[0085] refer to Figure 3 , which is a system architecture diagram of the image recognition method provided in an embodiment of the present application. The architecture at least includes a terminal device 301 and a server 302.

[0086] The terminal device 301 may be installed with multimedia applications, which may include applications associated with multimedia content, such as video playback applications, short video applications, and live broadcast applications. Types of multimedia applications include client applications, web applications, and mini-program applications. The terminal device 301 may include one or more processors 3011, memory 3012, an I / O interface 3013 for interacting with the server 302, and a display panel 3014. The terminal device 301 may be, but is not limited to, a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, or in-vehicle device.

[0087] The server 302 can be a background server for multimedia applications, providing corresponding services for multimedia applications. The server 302 can include one or more processors 3021, a memory 3022, and an I / O interface 3023 for interacting with the terminal device 301. In addition, the server 302 can also be configured with a database 3024. The server 302 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 301 and the server 302 can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0088] The image recognition method in the embodiment of the present application can be executed by the terminal device 301 or the server 302.

[0089] In the first case, the image recognition method may be executed by the terminal device 301 .

[0090] Terminal device 301 acquires an image to be recognized and detects at least one target object in the image to be recognized, obtaining identification frames corresponding to the at least one target object and determining partial incomplete recognition results for each identification frame. A target analysis object is then determined from the at least one target object based on the area of each identification frame. An overall recognition result for the target analysis object is then determined based on the partial incomplete recognition results for each identification frame corresponding to the target analysis object. Finally, an image recognition result for the image to be recognized is determined based on the overall recognition result for the target analysis object.

[0091] In the second case, the image recognition method may be executed by the server 302 .

[0092] Terminal device 301 obtains an image to be recognized and sends it to server 302. Server 302 detects at least one target object in the image to be recognized, obtains a corresponding identification frame for each target object, and determines a partial incomplete recognition result for each identification frame. Then, based on the area of each identification frame, it determines a target analysis object from the at least one target object. Based on the partial incomplete recognition results for each identification frame corresponding to the target analysis object, it determines an overall recognition result for the target analysis object. Finally, based on the overall recognition result for the target analysis object, it determines an image recognition result for the image to be recognized.

[0093] based on Figure 3 The system architecture diagram shown in FIG. 1 shows an embodiment of the present application, which provides a process of an image recognition method. Figure 4 As shown, the process of this method can be Figure 3 The terminal device or server shown executes the following steps:

[0094] Step S401 : detecting at least one target object in the image to be recognized, obtaining a recognition frame corresponding to the at least one target object, and determining a partial incomplete recognition result of each recognition frame.

[0095] Specifically, the type of target object can be a person, an animal, a plant, a building, etc. An image to be recognized can include one or more target objects. When an image to be recognized includes multiple target objects, the types of the multiple target objects can be exactly the same. For example, an image to be recognized includes three people. The types of the multiple target objects can also be different. For another example, an image to be recognized includes two people and a puppy.

[0096] A target object corresponds to one or more recognition boxes. A recognition box can cover part of the target object or the entire target object, and different recognition boxes can cover different areas of the target object. The partial part recognition result is used to indicate whether the target object or sub-target object within the recognition box is partially missing, that is, whether the target object or sub-target object within the recognition box is complete. A sub-target object is a partial area of the target object.

[0097] For example, if the target object is a human body, the body frame's missing part recognition result is used to indicate whether the body within the frame is missing parts, that is, whether the body within the frame is complete. If the sub-target object is a face, the face frame's missing part recognition result is used to indicate whether the face within the frame is missing parts, that is, whether the face within the frame is complete.

[0098] In a specific implementation, the recognition result probability and / or the recognition result label can be used to represent the part incomplete recognition result.

[0099] For example, Figure 5 As shown, the image to be identified includes two persons, namely person A and person B. By detecting the persons in the image to be identified, a human body recognition frame 501 corresponding to person A and a human body recognition frame 502 corresponding to person B are obtained. Among them, the incomplete recognition result of the human body recognition frame 501 is that the human body is complete, and the incomplete recognition result of the human body recognition frame 502 is that the human body is incomplete.

[0100] Step S402 : determining a target analysis object from at least one target object based on the area of each recognition frame.

[0101] Specifically, the area of the recognition box is the product of the length and width of the recognition box. For example, suppose Figure 5The length of the human body recognition frame A shown is 10 cm and the width is 5 cm, so the coverage area of the human body recognition frame A is 50 cm. 2 .

[0102] An identification frame whose area meets the preset conditions is selected from each identification frame, and then the target object corresponding to the selected identification frame is used as the target analysis object. The target analysis object represents the main target object in the image to be identified. The preset conditions can be set according to actual conditions.

[0103] Step S403 : determining an overall recognition result of the target analysis object based on the partial recognition results of each recognition frame corresponding to the target analysis object.

[0104] Step S404: determining an image recognition result of the image to be recognized based on the overall recognition result of the target analysis object.

[0105] Specifically, the overall recognition result of the target object is determined based on the partial incomplete recognition results of each identification frame corresponding to the target object. The overall recognition result of the target object can be divided into complete object and incomplete object. For different cropping directions, incomplete object can be further divided into left and right cropping, top cropping, and bottom cropping.

[0106] The overall recognition result of the target analysis object is used to represent the image recognition result of the entire image to be recognized. If the overall recognition result of the target analysis object is that the object is complete, the image recognition result of the image to be recognized is determined to be complete. If the overall recognition result of the target analysis object is that the object is incomplete, the image recognition result of the image to be recognized is determined to be incomplete.

[0107] In an embodiment of the present application, based on the area of the identification frame corresponding to at least one target object, the target analysis object is determined from at least one target object, and then based on the overall recognition result of the target analysis object, the image recognition result of the image to be identified is directly determined, without the need to identify the overall recognition result of each target object in the image, thereby improving the accuracy and efficiency of identifying whether there is an object incomplete problem in the image. Secondly, based on the partial incomplete recognition results of each identification frame corresponding to the target analysis object, the overall recognition result of the target analysis object is determined, that is, based on the characteristics of the target analysis object in multiple aspects, the completeness of the target analysis object is determined, thereby improving the accuracy of identifying the completeness of the target analysis object, and further improving the accuracy and recall rate of detecting whether there is an object incomplete problem in the image to be identified.

[0108] Optionally, in step S401, at least one target object in the image to be recognized is detected using a trained object detection model to obtain at least one recognition frame, and a partial incomplete recognition result of the at least one recognition frame is determined. The sample image used to train the object detection model is obtained by randomly cropping an image from which the target object has been segmented. The at least one recognition frame is then grouped based on a positional relationship between the at least one recognition frame to obtain recognition frames corresponding to the at least one target object.

[0109] Specifically, the target detection model can be an EfficientDet model, a YOLO (You Only Look Once) model, an SSD (Single Shot MultiBox Detector) model, an R-CNN (Regions with CNN features) model, or other detection models. Optionally, in an embodiment of the present application, the trained target detection model can be stored on the blockchain.

[0110] When preparing sample images for training the object detection model, we first obtain initial images that have been segmented to identify the various parts of each target object. During the segmentation process, these parts have been annotated pixel-wise. These initial images can be publicly available segmentation training data. These initial images are then randomly cropped to obtain sample images. Based on the existing segmentation annotations, the positions and sizes of the various recognition boxes in the sample images are determined. Finally, the incomplete labels for the parts of each recognition box are marked.

[0111] Taking the target object as human body, the initial image is as follows Figure 6 As shown in FIG, the initial image includes the body parts of the segmented person A. Figure 6 After random cropping of the initial image shown, we get Figure 7a The first sample image shown and Figure 7b In the second sample image shown, the first sample image includes the segmentation annotation of the left half of person A. Based on the segmentation annotation of the left half of person A, an identification frame 701 is determined, and the part defect label of identification frame 701 is marked as left and right human body cropping. The second sample image includes the segmentation annotation of the lower half of person A. Based on the segmentation annotation of the lower half of person A, an identification frame 702 is determined, and the part defect label of identification frame 702 is marked as upper human body cropping.

[0112] It should be noted that the method for constructing training data for training the target detection model in the embodiment of the present application is not limited to the method described above, and methods such as manual labeling, target detection, key point detection and human body segmentation can also be used.

[0113] Using a trained object detection model, at least one target object in the image to be identified is detected. After obtaining at least one identification frame, the system then determines whether there is any intersection between the identification frames and the degree of intersection between them based on their positional relationships. The degree of intersection can refer to the intersection area between the identification frames or the intersection-over-union ratio between the identification frames. Frames with an intersection greater than a preset threshold are then identified as the same target object. Frames belonging to the same target object are then grouped together to obtain the corresponding part identification frames for each target object.

[0114] In an embodiment of the present application, sample images for training the target detection model are obtained by randomly cropping the currently disclosed human body segmentation training data, and the position, size and part defect labels of the recognition box in the sample image are determined based on the existing segmentation annotations, without the need to manually annotate the position, size and part defect labels of the recognition box, thereby greatly reducing the cost of sample annotation and improving the efficiency of sample annotation.

[0115] In one possible implementation, the at least one recognition frame includes at least one target object recognition frame, and the partial incomplete recognition result of the at least one recognition frame includes a target object recognition result, where the target object recognition result indicates whether the target object within the target object recognition frame is complete. Using a trained object detection model, at least one target object in the image to be recognized is detected to obtain at least one target object recognition frame, and a target object recognition result for the at least one target object recognition frame is determined.

[0116] Specifically, each target object corresponds to a target object recognition box. The trained object detection model performs a comprehensive detection of the target object in the image to be recognized, generating the target object recognition box. The target object recognition results for the target object recognition box include both complete and incomplete target objects. Incomplete target objects can be further categorized into different types of missing parts based on different cropping directions.

[0117] In a specific implementation, when the cropping direction is left-right, the target object recognition frame only includes the left or right half of the target object. Since the target object is generally bilaterally symmetrical, the part defect condition can be set to left-right cropping of the target object. When the cropping direction is up-down, the target object recognition frame only includes the upper or lower half of the target object. When the target object recognition frame includes the upper half of the target object, the part defect condition is set to bottom cropping of the target object. When the target object recognition frame includes the lower half of the target object, the part defect condition is set to top cropping of the target object.

[0118] For the target object recognition box, the trained target detection model is used to determine the target object recognition result probabilities of the target object recognition box being the complete target object, the target object cropped left and right, the target object cropped top, and the target object cropped bottom. The target object recognition result with the highest probability is taken as the target object recognition result of the target object recognition box.

[0119] For example, Figure 8a As shown, the object detection model is used to detect the person in the image to be identified, and two human body recognition frames are obtained, namely human body recognition frame 801 and human body recognition frame 802. Human body recognition frame 801 includes the entire human body, that is, the human body recognition result of human body recognition frame 801 is a complete human body. Human body recognition frame 802 includes the lower half of the human body, that is, the human body recognition result of human body recognition frame 802 is a cropped upper part of the human body.

[0120] In one possible implementation, at least one recognition frame includes at least one sub-target object recognition frame, where the sub-target object is a partial area of the target object. The partial incomplete recognition result of the at least one recognition frame includes a sub-target object recognition result, where the sub-target object recognition result indicates whether the sub-target object within the sub-target object recognition frame is complete. Using a trained target detection model, at least one target object in the image to be recognized is detected to obtain at least one sub-target object recognition frame, and a sub-target object recognition result for the at least one sub-target object recognition frame is determined.

[0121] For example, when the target object is a human body, the sub-target object recognition frame includes at least one of the following: a face recognition frame and a head recognition frame. The sub-target object recognition result of the sub-target object recognition frame includes a complete sub-target object and an incomplete sub-target object.

[0122] For different cropping directions, the incomplete sub-target objects can be further subdivided into different categories of part defects.

[0123] In a specific implementation, when the cropping direction is left-right, the sub-target object identification frame only includes the left or right half of the sub-target object. Since sub-target objects are generally bilaterally symmetrical, the part defect condition can be set to left-right cropping of the sub-target object. When the cropping direction is up-down, the sub-target object identification frame only includes the upper or lower half of the sub-target object. When the sub-target object identification frame includes the upper half of the sub-target object, the part defect condition is set to bottom cropping of the sub-target object. When the sub-target object identification frame includes the lower half of the sub-target object, the part defect condition is set to top cropping of the sub-target object.

[0124] For the sub-target object recognition frame, the trained target detection model is used to determine the probabilities of the sub-target object recognition results when the sub-target object recognition frame is the complete sub-target object, the left and right sub-target object cropped, the upper part of the sub-target object cropped, and the lower part of the sub-target object cropped. The sub-target object recognition result with the largest probability is used as the sub-target object recognition result of the sub-target object recognition frame.

[0125] For example, Figure 8b As shown, the object detection model is used to detect the person in the image to be recognized, and two facial recognition frames are obtained, namely facial recognition frame 803 and facial recognition frame 804. Facial recognition frame 803 includes the entire face, that is, the facial recognition result of facial recognition frame 803 is a complete face. Facial recognition frame 804 includes the left half of the face, that is, the facial recognition result of facial recognition frame 804 is a left-right cropped face.

[0126] Optionally, at least one target object in the image to be identified is detected using a trained target detection model. After obtaining at least one sub-target object identification frame, whether there is an intersection between the sub-target object identification frames and the degree of intersection between the sub-target object identification frames are determined based on the positional relationship between the sub-target object identification frames. Sub-target object identification frames with an intersection degree greater than a preset threshold are then determined to be sub-target object identification frames of the same target object. The sub-target object identification frames of the same target object are then grouped together to obtain sub-target object identification frames corresponding to each target object.

[0127] In one possible implementation, at least one identification frame includes at least one target object identification frame and at least one sub-target object identification frame, the sub-target object is a partial area of the target object, and the partial incomplete identification result of at least one identification frame includes a target object identification result and a sub-target object identification result, wherein the target object identification result is used to characterize whether the target object within the target object identification frame is complete, and the sub-target object identification result is used to characterize whether the sub-target object within the sub-target object identification frame is complete.

[0128] Using a trained target detection model, at least one target object in an image to be identified is detected to obtain at least one target object identification frame and at least one sub-target object identification frame, and a target object recognition result of at least one target object identification frame and a sub-target object recognition result of at least one sub-target object identification frame are determined.

[0129] Specifically, each target object corresponds to a target object recognition frame and at least one sub-target object recognition frame. The trained target detection model detects the target object in the image to be recognized as a whole to obtain the target object recognition frame. The trained target detection model also detects a portion of the target object in the image to be recognized to obtain the sub-target object recognition frame.

[0130] For example, if the target object is a human body, the target object recognition frame is a human body recognition frame, and the sub-target object recognition frames include at least one of the following: a face recognition frame and a head recognition frame. The target object recognition results for the target object recognition frame include either a complete target object or an incomplete target object, and the sub-target object recognition results for the sub-target object recognition frame include either a complete sub-target object or an incomplete sub-target object.

[0131] For different cropping directions, the incompleteness of the target object can be further subdivided into different categories of part defects. Similarly, the incompleteness of the sub-target object can also be further subdivided into different categories of part defects. The specific subdivisions have been introduced in the previous article and will not be repeated here.

[0132] For example, taking the target object as a human body and the sub-target objects as the face and the head, Figure 9 The identification frames corresponding to the human body, face and head, as well as the incomplete recognition results of the parts of the identification frames are exemplarily shown.

[0133] exist Figure 9 In the figure, the human body recognition result of human body recognition frame 901 is that the human body is complete, and the probability of human body recognition result is 0.91; the human body recognition result of human body recognition frame 902 is that the human body is cropped from left to right, and the probability of human body recognition result is 0.68; the human body recognition result of human body recognition frame 903 is that the upper part of the human body is cropped, and the probability of human body recognition result is 0.75; under normal circumstances, an image or video may also only include the upper half of the human body. For example, most live broadcast videos only include the upper body of the anchor. Therefore, the situation where the lower part of the human body is cropped is classified as the situation where the human body is complete.

[0134] The head recognition result of head recognition frame 904 is that the head is complete, and the probability of head recognition result is 0.90; the head recognition result of head recognition frame 905 is that the head is cropped left and right, and the probability of head recognition result is 0.83; the head recognition result of head recognition frame 906 is that the upper part of the head is cropped, and the probability of head recognition result is 0.69; the head recognition result of head recognition frame 907 is that the lower part of the head is cropped, and the probability of head recognition result is 0.77.

[0135] The facial recognition result of facial recognition frame 908 is that the face is complete, and the facial recognition result probability is 0.90; the facial recognition result of facial recognition frame 909 is that the face is cropped left and right, and the facial recognition result probability is 0.69; the facial recognition result of facial recognition frame 910 is that the upper part of the face is cropped, and the facial recognition result probability is 0.78; the facial recognition result of facial recognition frame 911 is that the lower part of the face is cropped, and the facial recognition result probability is 0.70.

[0136] Optionally, at least one target object in the image to be identified is detected using a trained target detection model. After obtaining at least one target object identification frame and at least one sub-target object identification frame, it is determined whether there is an intersection between the identification frames and the degree of intersection between the identification frames based on the positional relationship between the identification frames. The identification frames with an intersection degree greater than a preset threshold are then determined as identification frames of the same target object. The identification frames of the same target object are then grouped together to obtain identification frames corresponding to each target object.

[0137] For example, Figure 10 As shown, the human body in the image to be identified is detected by the target detection model, and 6 recognition frames are obtained, namely, human body recognition frame 1001, human body recognition frame 1002, head recognition frame 1003, head recognition frame 1004, face recognition frame 1005 and face recognition frame 1006. Among them, the degree of intersection between the human body recognition frame 1001, the head recognition frame 1003 and the face recognition frame 1005 is greater than the preset threshold, and the degree of intersection between the human body recognition frame 1002, the head recognition frame 1004 and the face recognition frame 1006 is greater than the preset threshold. Then, the human body recognition frame 1001, the head recognition frame 1003 and the face recognition frame 1005 are grouped together to obtain a target recognition frame set A, and the human body recognition frame 1002, the head recognition frame 1004 and the face recognition frame 1006 are grouped together to obtain a target recognition frame set B.

[0138] In an embodiment of the present application, through the target detection model, the target object recognition frame and the sub-target object recognition frame are detected, and the target object recognition result of the target object recognition frame and the sub-target object recognition result of the sub-target object recognition frame are determined, thereby realizing multi-faceted recognition and detection of the target object, making the obtained target object features more comprehensive, thereby improving the accuracy of identifying incomplete target objects.

[0139] Optionally, since at least one target object in the image to be identified is detected by the target detection model, the obtained recognition frame may contain redundant recognition frames. In order to facilitate the subsequent incomplete determination of the target object in the image to be identified based on the recognition frame, the recognition frame needs to be screened to remove redundant recognition frames. Secondly, since in this application, the recognition frame obtained by the target detection model includes a target object recognition frame and a sub-target object recognition frame, and the sub-target object may also include multiple types, in order to avoid mistakenly filtering out valid recognition frames due to the existence of intersections between the target object recognition frame and the sub-target object recognition frame, or the existence of intersections between different types of sub-target object recognition frames, a category-sensitive non-maximum suppression algorithm is proposed in an embodiment of the present application to screen the target object recognition frame and different types of sub-target object recognition frames respectively.

[0140] Specifically, the screening operation is iteratively performed on the first screening object until the first screening object meets the first preset condition, wherein the initial first screening object is at least one target object identification frame, and each iterative process includes the following screening operations:

[0141] A first analysis benchmark identification frame with the highest probability of a target object recognition result is obtained from the first screening object. A first intersection-over-union (IoU) ratio (IOR) is then determined between the first analysis benchmark identification frame and each of the other target object identification frames in the first screening object. Target object identification frames whose IOU ratios exceed a first threshold are removed from the other target object identification frames. The first screening object is then updated to the target object identification frame that remains within the other target object identification frames.

[0142] Specifically, the intersection-to-union ratio is a ratio of the intersection to the union, and the first preset condition may be that there is no target object identification frame in the first screening object.

[0143] For example, if the target object is the human body, Figure 11 As shown, it is assumed that the image to be identified includes 6 human body recognition frames, namely human body recognition frame A (human body recognition result probability is 0.89), human body recognition frame B (human body recognition result probability is 0.69), human body recognition frame C (human body recognition result probability is 0.78), human body recognition frame D (human body recognition result probability is 0.90), human body recognition frame E (human body recognition result probability is 0.88), and human body recognition frame F (human body recognition result probability is 0.70).

[0144] From the six human recognition frames, the first analysis benchmark recognition frame with the highest probability of human recognition is determined to be human recognition frame D. The first intersection-over-union (IoU) ratios are then calculated for human recognition frame D with human recognition frames A, B, C, E, and F. Since the IoU ratios for human recognition frame D with human recognition frames A, B, and C are not greater than a first threshold, human recognition frames A, B, and C are retained. Since the IoU ratios for human recognition frame D with human recognition frames E and F are greater than the first threshold, human recognition frames E and F are removed.

[0145] From the three retained human frames, the first analysis benchmark frame with the highest probability of human recognition is determined to be human frame A. The first intersection-over-union (IoU) ratios are then calculated for human frame A, human frame B, and human frame C. Since the IoU ratios for human frame A, human frame B, and human frame C are all greater than the first threshold, human frame B and human frame C are removed. Since no human frames are retained, the screening process ends.

[0146] Optionally, the screening operation is iteratively performed on the second screening object until the second screening object meets the second preset condition, wherein the initial second screening object is at least one sub-target object identification frame, and each iterative process includes the following screening operations:

[0147] A second analysis benchmark identification frame with the highest probability of a sub-target object recognition result is obtained from the second screening object. A second intersection-over-union (IoU) ratio (IOR) is then determined between the second analysis benchmark identification frame and each of the other sub-target object identification frames in the second screening object. Sub-target object identification frames whose second IORs exceed a second threshold value are removed from the other sub-target object identification frames. The second screening object is updated to the sub-target object identification frames retained from the other sub-target object identification frames.

[0148] Specifically, the sub-target object recognition frames included in the initial second screening object are sub-target object recognition frames of the same type, for example, all are face recognition frames, or all are head recognition frames.

[0149] For example, if the sub-target object is a face, Figure 12 As shown, it is assumed that the image to be recognized includes 6 face recognition frames, namely face recognition frame a1 (face recognition result probability is 0.87), face recognition frame b1 (face recognition result probability is 0.93), face recognition frame c1 (face recognition result probability is 0.78), face recognition frame d1 (face recognition result probability is 0.86), face recognition frame e1 (face recognition result probability is 0.69), and face recognition frame f1 (face recognition result probability is 0.76).

[0150] From the six facial recognition frames, the second analysis reference frame with the highest probability of facial recognition is determined to be facial recognition frame b1. The second intersection-over-union (IoU) ratios are then calculated for facial recognition frame b1 with facial recognition frame a1, facial recognition frame c1, facial recognition frame d1, facial recognition frame e1, and facial recognition frame f1. Since the IoU ratios for facial recognition frame b1, facial recognition frame d1, facial recognition frame e1, and facial recognition frame f1 are not greater than a second threshold, facial recognition frames d1, e1, and f1 are retained. Since the IoU ratios for facial recognition frame b1, facial recognition frame a1, and facial recognition frame c1 are greater than the second threshold, facial recognition frames a1 and c1 are removed.

[0151] From the three retained facial recognition frames, the second analysis reference frame with the highest probability of facial recognition is determined to be facial recognition frame d1. The second intersection-over-union (IoU) ratios are then calculated for facial recognition frame d1, facial recognition frame e1, and facial recognition frame f1. Since the IoU ratios for facial recognition frame d1, facial recognition frame e1, and facial recognition frame f1 are all greater than the second threshold, facial recognition frames e1 and f1 are removed. Since no facial recognition frames remain, the screening process ends.

[0152] In an embodiment of the present application, the target object recognition frame and different types of sub-target object recognition frames are screened separately to avoid mistakenly filtering out valid recognition frames due to intersections between the target object recognition frame and the sub-target object recognition frames, or intersections between different types of sub-target object recognition frames, thereby improving the accuracy of recognition frame screening and further improving the accuracy of subsequent determination of incompleteness of the target object in the image to be identified based on the recognition frame.

[0153] Optionally, in the above step S402, when the area of the recognition frame is larger, the target object in the recognition frame occupies a larger area in the image to be recognized, and the user is more likely to notice the target object when viewing the image. The target object can also be referred to as the main target object in the image to be recognized. When the user views the image to be recognized, if the main target object in the image to be recognized is complete, while other secondary target objects are incomplete, it will not affect the user's viewing experience. In view of this, the embodiment of the present application selects the target object corresponding to the recognition frame with the largest area as the target analysis object.

[0154] Specifically, for each identification box corresponding to each target object in the at least one target object, perform the following operations:

[0155] For each recognition frame corresponding to a target object, obtain the candidate recognition frame with the largest area from each recognition frame corresponding to the target object. Then, select the candidate recognition frame with the largest area from the obtained candidate recognition frames as the target recognition frame. The target object corresponding to the target recognition frame in at least one target object is used as the target analysis object.

[0156] Specifically, each recognition frame corresponding to a target object may include only the target object recognition frame, only the sub-target object recognition frame, or both. First, the candidate recognition frame with the largest area is selected from each recognition frame corresponding to the target object. Then, the target recognition frame with the largest area is selected from each candidate recognition frame. The target object within the target recognition frame is regarded as the target analysis object, i.e., the primary target object in the image to be recognized.

[0157] For example, if Figure 13 As shown, the image to be recognized includes two persons, namely person M and person N. The recognition frames corresponding to person M include a body recognition frame 1301, a head recognition frame 1302, and a face recognition frame 1303. The recognition frames corresponding to person N include a body recognition frame 1304, a head recognition frame 1305, and a face recognition frame 1306.

[0158] A comparison reveals that among body recognition frame 1301, head recognition frame 1302, and face recognition frame 1303, body recognition frame 1301 has the largest area. Body recognition frame 1301 is then selected as a candidate recognition frame. A comparison reveals that among body recognition frame 1304, head recognition frame 1305, and face recognition frame 1306, body recognition frame 1304 has the largest area. Body recognition frame 1304 is then selected as a candidate recognition frame. The areas of body recognition frame 1301 and body recognition frame 1304 are then compared. If body recognition frame 1301 is larger, body recognition frame 1301 is selected as the target recognition frame, and person M is selected as the target analysis object.

[0159] It should be noted that the embodiments of the present application are not limited to the above-mentioned method of determining the target analysis object, but may also be other implementation methods, such as merging the identification boxes corresponding to each target object into a total set, and then directly determining the target identification box with the largest area from the total set, and using the target object corresponding to the target identification box as the target analysis object.

[0160] In an embodiment of the present application, the identification box with the largest area is obtained from the identification boxes corresponding to each target object, and then the target object corresponding to the identification box with the largest area is used as the target analysis object. Based on the overall recognition result of the target analysis object, the image recognition result of the image to be identified is directly determined. Without the need to identify the overall recognition result of each target object in the image, the problem of incomplete identification of target objects when there are multiple target objects in the image is effectively solved, and at the same time, the accuracy and efficiency of identifying incomplete target objects in the image are improved.

[0161] Optionally, in the above step S403, if the part incomplete recognition results of each recognition frame corresponding to the target analysis object all indicate that the part is not incomplete, then the overall recognition result of the target analysis object is determined to be a complete object; otherwise, the overall recognition result of the target analysis object is determined to be an incomplete object.

[0162] Specifically, the incomplete part recognition result of the recognition frame represents that the part is not incomplete, which means that the target object or sub-target object in the recognition frame is complete.

[0163] In one possible implementation, the identification frame corresponding to the target object only includes the target object identification frame. When the target object identification result of the target object identification frame indicates that the target object is complete, the overall identification result of the target analysis object is determined to be complete. If the target object identification result of the target object identification frame indicates any of the following: the target object is cropped left and right, the target object is cropped from the bottom, or the target object is cropped from the top, the overall identification result of the target analysis object is determined to be incomplete.

[0164] In one possible implementation, the identification frame corresponding to the target object only includes the sub-target object identification frame. When the sub-target object identification results of each sub-target object identification frame are all complete sub-target objects, the overall identification result of the target analysis object is determined to be a complete object. If at least one of the following items exists in the sub-target object identification results of each sub-target object identification frame: the sub-target object is cropped left and right, the sub-target object is cropped at the bottom, and the sub-target object is cropped at the top, the overall identification result of the target analysis object is determined to be an incomplete object.

[0165] For example, assume that the image to be identified includes two persons, Person X and Person Y, where Person X is the target person for analysis. The identification frame corresponding to the target person includes a head identification frame and a face identification frame. If the head identification result of the head identification frame is a cropped upper head, and the face identification result of the face identification frame is a complete face, then the overall identification result of Person X is determined to be incomplete, and the image to be identified is therefore determined to be incomplete.

[0166] In one possible implementation, the identification frame corresponding to the target object includes both a target object identification frame and a sub-target object identification frame. When the target object identification result of the target object identification frame indicates that the target object is complete, and the sub-target object identification results of each sub-target object identification frame indicate that the sub-target object is complete, the overall identification result of the target analysis object is determined to be complete. If the target object identification frame's target object identification result is any of the following: the target object is cropped left and right, the target object is cropped below, or the target object is cropped above, or if the sub-target object identification results of each sub-target object identification frame show at least one of the following: the sub-target object is cropped left and right, the sub-target object is cropped below, or the sub-target object is cropped above, the overall identification result of the target analysis object is determined to be incomplete.

[0167] For example, assume that the image to be identified includes two persons, namely Person X and Person Y, where Person X is the target person for analysis. The target recognition frame set corresponding to the target person for analysis includes a body recognition frame, a head recognition frame, and a face recognition frame.

[0168] If the human body recognition result of the human body recognition frame is that the human body is complete, the head recognition result of the head recognition frame is that the head is complete, and the face recognition result of the face recognition frame is that the face is complete, then the overall recognition result of person X is determined to be complete, and the image to be recognized can be determined to be a complete image.

[0169] If the human body recognition result of the human body recognition frame is that the upper part of the human body is cropped, the head recognition result of the head recognition frame is that the head is complete, and the face recognition result of the face recognition frame is that the face is complete, then the overall recognition result of the person X is determined to be incomplete, and the image to be recognized can be determined to be an incomplete image.

[0170] In an embodiment of the present application, the overall recognition result of the target analysis object is determined based on the partial incomplete recognition results of each recognition frame corresponding to the target analysis object, that is, the completeness of the target analysis object is determined through the characteristics of the target analysis object in multiple aspects, thereby improving the accuracy of recognizing the incompleteness of the target object in the image.

[0171] Optionally, in the above step S403, if the incomplete recognition results of the parts of each identification frame corresponding to the target analysis object all indicate that the object is not incomplete, and the probabilities of the recognition results indicating that the object is not incomplete are all greater than the corresponding judgment threshold, then the overall recognition result of the target analysis object is determined to be a complete object; otherwise, the overall recognition result of the target analysis object is determined to be an incomplete object.

[0172] Specifically, decision thresholds are pre-set for the target object recognition frame and each type of sub-target object recognition frame. A recognition result indicating partial incompleteness indicates that the target object or sub-target object within the recognition frame is complete. The probability of a recognition result indicating completeness indicates the probability that the target object or sub-target object within the recognition frame is complete.

[0173] In one possible implementation, the identification frame corresponding to the target object only includes the target object identification frame. When the target object recognition result for the target object identification frame indicates that the target object is complete, and the probability of the target object being complete is greater than a corresponding decision threshold, the overall recognition result for the target analysis object is determined to be complete; otherwise, the overall recognition result for the target analysis object is determined to be incomplete.

[0174] In one possible implementation, the identification frame corresponding to the target object includes only the sub-target object identification frame. If the sub-target object identification results for each sub-target object identification frame are all complete, and the probability of the sub-target object being complete is greater than a corresponding decision threshold, the overall identification result for the target analysis object is determined to be complete. Otherwise, the overall identification result for the target analysis object is determined to be incomplete.

[0175] For example, assume that the image to be identified includes two persons, namely person X and person Y, wherein person X is the target person for analysis. The target recognition frame corresponding to the target person for analysis includes a head recognition frame and a face recognition frame.

[0176] If the head recognition result of the head recognition frame is that the upper part of the head is cropped, and the face recognition result of the face recognition frame is that the face is complete, and the probability of the face being complete is greater than the face decision threshold (0.7), then the overall recognition result of person X is determined to be incomplete, and the image to be recognized can be determined to be an incomplete image.

[0177] If the head recognition result of the head recognition frame is that the head is complete, and the probability of the head being complete is greater than the head judgment threshold (0.7), and the face recognition result of the face recognition frame is that the face is complete, and the probability of the face being complete is greater than the face judgment threshold (0.7), then the overall recognition result of person X is determined to be complete, and the image to be recognized can be determined to be a complete image.

[0178] In one possible implementation, the identification frame corresponding to the target object includes both a target object identification frame and sub-target object identification frames. When the target object identification result for the target object identification frame indicates that the target object is complete, the sub-target object identification results for each sub-target object identification frame indicate that the sub-target objects are complete, and both the probability of the target object being complete and the probability of each sub-target object being complete are greater than corresponding decision thresholds, the overall identification result for the target analysis object is determined to be complete. Otherwise, the overall identification result for the target analysis object is determined to be incomplete.

[0179] For example, assume that the image to be identified includes two persons, namely Person X and Person Y, where Person X is the target person for analysis. The identification frames corresponding to the target person for analysis include a body identification frame, a head identification frame, and a face identification frame.

[0180] If the human body recognition result of the human body recognition frame is that the human body is complete, and the probability of the human body being complete is greater than the human body judgment threshold (0.6), the head recognition result of the head recognition frame is that the head is complete, and the probability of the head being complete is greater than the head judgment threshold (0.7), and the face recognition result of the face recognition frame is that the face is complete, and the probability of the face being complete is greater than the face judgment threshold (0.7), then the overall recognition result of person X is determined to be complete, and the image to be recognized can be determined to be a complete image.

[0181] If the human body recognition result of the human body recognition frame is that the upper part of the human body is cropped, the head recognition result of the head recognition frame is that the head is complete, and the probability of the head being complete is greater than the head judgment threshold (0.7), and the face recognition result of the face recognition frame is that the face is complete, and the probability of the face being complete is greater than the face judgment threshold (0.7), then the overall recognition result of person X is determined to be incomplete, and the image to be recognized can be determined to be an incomplete image.

[0182] In an embodiment of the present application, the overall recognition result of the target analysis object is determined based on the incomplete recognition results of the parts of each recognition frame corresponding to the target analysis object and the recognition result probability, that is, the completeness of the target analysis object is determined through the characteristics of the target analysis object in multiple aspects, thereby improving the accuracy of recognizing the incompleteness of the target object in the image.

[0183] In order to better explain the embodiment of the present application, the following takes the image or video review scenario as an example to introduce an image recognition method provided by the embodiment of the present application, which is executed by the server, such as Figure 14 As shown, the following steps are included:

[0184] Image A uploaded or published by a user is obtained. Image A can be an image from a video or an image uploaded directly by the user. A trained object detection model is used to detect at least one person in image A, obtaining a person recognition frame 1401, a head recognition frame 1402, a face recognition frame 1403, and a face recognition frame 1404. The person recognition result for person recognition frame 1401 is a complete person, with a corresponding person recognition probability of 0.806. The head recognition result for head recognition frame 1402 is a complete head, with a corresponding head recognition probability of 0.841. The face recognition result for face recognition frame 1403 is a complete face, with a corresponding face recognition probability of 0.984. The face recognition result for face recognition frame 1404 is a cropped face, with a corresponding face recognition probability of 0.362.

[0185] If the degree of intersection between human frame 1401, head frame 1402, and facial frame 1403 exceeds a preset threshold, human frame 1401, head frame 1402, and facial frame 1403 are assigned to a single frame set, and facial frame 1404 is assigned to a separate frame set. The areas of human frame 1401, head frame 1402, and facial frame 1403 are compared to determine that human frame 1401 has the largest area. The areas of human frame 1401 and facial frame 1404 are then compared to determine that human frame 1401 has the largest area. The person within human frame 1401 is then selected as the target person for analysis.

[0186] Since the human body recognition result of human body recognition frame 1401 corresponding to the target human body is complete, the head recognition result of head recognition frame 1402 corresponding to the target human body is complete, and the face recognition result of face recognition frame 1403 corresponding to the target human body is complete, the overall recognition result of the target human body is determined to be complete, and thus image A can be determined to be a complete image. Image A is placed in the recommendation pool so that content related to image A can be recommended to other users later.

[0187] In an embodiment of the present application, based on the area of each identification box corresponding to at least one target object, the target analysis object is determined from at least one target object, and then based on the overall recognition result of the target analysis object, the image recognition result of the image to be identified is directly determined, without the need to identify the overall recognition result of each target object in the image, thereby improving the accuracy and efficiency in identifying whether there is an object incomplete problem in the image. Secondly, based on the partial incomplete recognition results of each identification box corresponding to the target analysis object, the overall recognition result of the target analysis object is determined, that is, the completeness of the target analysis object is determined through the characteristics of the target analysis object in multiple aspects, thereby improving the accuracy of identifying the completeness of the target analysis object, and then improving the accuracy and recall rate of detecting whether there is an object incomplete problem in the image to be identified. After applying the image recognition method in the embodiment of the present application to the cover image selection and content review of information flow products, the proportion of incomplete cover image problems in the product has dropped significantly, thereby improving the user experience.

[0188] Based on the same technical concept, the embodiment of the present application provides a structural diagram of an image recognition device, such as Figure 15 As shown, the apparatus 1500 includes:

[0189] The detection module 1501 is configured to detect at least one target object in the image to be recognized, obtain recognition frames corresponding to the at least one target object, and determine a partial recognition result of each recognition frame;

[0190] A selection module 1502 is configured to determine a target analysis object from the at least one target object based on the areas of the respective identification frames;

[0191] Matching model 1503, used to determine the overall recognition result of the target analysis object based on the partial recognition results of each recognition frame corresponding to the target analysis object;

[0192] The discrimination module 1504 is configured to determine an image recognition result of the image to be recognized based on the overall recognition result of the target analysis object.

[0193] Optionally, the detection module 1501 is specifically configured to:

[0194] Using a trained target detection model, at least one target object in the image to be identified is detected to obtain at least one identification frame, and a partial incomplete identification result of the at least one identification frame is determined, wherein the sample image used for training the target detection model is obtained by randomly cropping an image from which the target object has been segmented;

[0195] Based on the positional relationship between the at least one recognition frame, the at least one recognition frame is grouped to obtain recognition frames corresponding to the at least one target object.

[0196] Optionally, the at least one recognition frame includes at least one target object recognition frame and at least one sub-target object recognition frame, the sub-target object being a partial area of the target object, and the partial incomplete recognition result of the at least one recognition frame includes a target object recognition result and a sub-target object recognition result, wherein the target object recognition result is used to indicate whether the target object within the target object recognition frame is complete, and the sub-target object recognition result is used to indicate whether the sub-target object within the sub-target object recognition frame is complete;

[0197] The detection module 1501 is specifically used for:

[0198] At least one target object in the image to be identified is detected using the trained target detection model to obtain at least one target object identification frame and at least one sub-target object identification frame, and a target object recognition result of the at least one target object identification frame and a sub-target object recognition result of the at least one sub-target object identification frame are determined.

[0199] Optionally, the target object recognition frame is a human body recognition frame, and the sub-target object recognition frame includes at least one of the following: a face recognition frame and a head recognition frame.

[0200] Optionally, the detection module 1501 is further configured to:

[0201] The at least one identification frame is grouped based on a positional relationship between the at least one identification frame. Before obtaining identification frames corresponding to the at least one target object, a screening operation is iteratively performed on a first screening object until the first screening object satisfies a first preset condition, wherein the initial first screening object is the at least one target object identification frame, and each iterative process includes the following screening operations:

[0202] Obtaining a first analysis reference recognition frame with the highest probability of a target object recognition result from the first screening objects;

[0203] Determine a first intersection-over-union ratio between the first analysis reference recognition frame and each of the other target object recognition frames in the first screening object;

[0204] Remove the target object recognition frames whose first intersection-over-union ratio is greater than a first threshold from the other target object recognition frames;

[0205] The first screening object is updated to the target object recognition frame retained in the other target object recognition frames.

[0206] Optionally, the detection module 1501 is further configured to:

[0207] Iteratively perform a screening operation on the second screening object until the second screening object satisfies a second preset condition, wherein the initial second screening object is the at least one sub-target object identification frame, and each iterative process includes the following screening operations:

[0208] Obtaining a second analysis reference recognition frame with the highest probability of sub-target object recognition result from the second screening objects;

[0209] Determine a second intersection-over-union ratio between the second analysis reference recognition frame and each of the other sub-target object recognition frames in the second screening object;

[0210] Remove the sub-target object identification frames whose second intersection-over-union ratio is greater than a second threshold from the other sub-target object identification frames;

[0211] The second screening object is updated to the sub-target object identification frame retained in each of the other sub-target object identification frames.

[0212] Optionally, the selection module 1502 is specifically configured to:

[0213] For each identification frame corresponding to each target object in the at least one target object, perform the following operations:

[0214] For each recognition frame corresponding to a target object, obtaining a candidate recognition frame with the largest area from among the recognition frames corresponding to the target object;

[0215] From the obtained candidate recognition frames, select the candidate recognition frame with the largest area as the target recognition frame;

[0216] The target object corresponding to the target identification frame in at least one target object is used as a target analysis object.

[0217] Optionally, the matching module 1503 is specifically configured to:

[0218] If the part incomplete recognition results of each recognition frame corresponding to the target analysis object all indicate that the parts are not incomplete, then the overall recognition result of the target analysis object is determined to be a complete object; otherwise, the overall recognition result of the target analysis object is determined to be an incomplete object.

[0219] Optionally, the identification module 1504 is specifically configured to:

[0220] If the overall recognition result of the target analysis object is that the object is complete, determining that the image recognition result of the image to be recognized is a complete image;

[0221] If the overall recognition result of the target analysis object is that the object is incomplete, then it is determined that the image recognition result of the image to be recognized is an incomplete image.

[0222] In an embodiment of the present application, based on the area of each identification box corresponding to at least one target object, the target analysis object is determined from at least one target object, and then based on the overall recognition result of the target analysis object, the image recognition result of the image to be identified is directly determined, without the need to identify the overall recognition result of each target object in the image, thereby improving the accuracy and efficiency in identifying whether there is an object incomplete problem in the image. Secondly, based on the partial incomplete recognition results of each identification box corresponding to the target analysis object, the overall recognition result of the target analysis object is determined, that is, the completeness of the target analysis object is determined through the characteristics of the target analysis object in multiple aspects, thereby improving the accuracy of identifying the completeness of the target analysis object, and then improving the accuracy and recall rate of detecting whether there is an object incomplete problem in the image to be identified. After applying the image recognition method in the embodiment of the present application to the cover image selection and content review of information flow products, the proportion of incomplete cover image problems in the product has dropped significantly, thereby improving the user experience.

[0223] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 16 As shown, it includes at least one processor 1601 and a memory 1602 connected to the at least one processor. The specific connection medium between the processor 1601 and the memory 1602 is not limited in the embodiment of the present application. Figure 16For example, the processor 1601 and the memory 1602 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0224] In the embodiment of the present application, the memory 1602 stores instructions that can be executed by at least one processor 1601. The at least one processor 1601 can execute the steps included in the aforementioned image recognition method by executing the instructions stored in the memory 1602.

[0225] Among them, the processor 1601 is the control center of the computer device. It can use various interfaces and lines to connect various parts of the computer device. It can perform image recognition by running or executing instructions stored in the memory 1602 and calling data stored in the memory 1602. Optionally, the processor 1601 may include one or more processing units. The processor 1601 may integrate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1601. In some embodiments, the processor 1601 and the memory 1602 can be implemented on the same chip. In some embodiments, they can also be implemented on separate chips.

[0226] The processor 1601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0227] Memory 1602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. Memory 1602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0228] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned image recognition method.

[0229] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0230] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0231] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0232] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0233] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0234] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An image recognition method, characterized in that: include: At least one target object in the image to be identified is detected, identification frames corresponding to the at least one target object are obtained, and partial incomplete identification results of each identification frame are determined; the at least one identification frame includes at least one target object identification frame and at least one sub-target object identification frame, where the sub-target object is a partial area of the target object; the partial incomplete identification results of the at least one identification frame include a target object identification result and a sub-target object identification result, wherein the target object identification result is used to indicate whether the target object within the target object identification frame is complete, and the sub-target object identification result is used to indicate whether the sub-target object within the sub-target object identification frame is complete; determining a target analysis object from the at least one target object based on the areas of the respective recognition frames, the target analysis object representing a main target object in the image to be recognized; Determining an overall recognition result of the target analysis object based on the partial incomplete recognition results of each recognition frame corresponding to the target analysis object; the overall recognition result includes whether the object is complete or incomplete; An image recognition result of the image to be recognized is determined based on the overall recognition result of the target analysis object.

2. The method according to claim 1, wherein The detecting of at least one target object in the image to be identified, obtaining identification frames corresponding to the at least one target object, and determining the incomplete identification results of the parts of each identification frame include: Using a trained target detection model, at least one target object in the image to be identified is detected to obtain at least one identification frame, and a partial incomplete identification result of the at least one identification frame is determined, wherein the sample image used for training the target detection model is obtained by randomly cropping an image from which the target object has been segmented; Based on the positional relationship between the at least one recognition frame, the at least one recognition frame is grouped to obtain recognition frames corresponding to the at least one target object.

3. The method according to claim 2, wherein The detecting of at least one target object in the image to be identified by using the trained target detection model, obtaining at least one identification frame, and determining a partial incomplete identification result of the at least one identification frame include: At least one target object in the image to be identified is detected using the trained target detection model to obtain at least one target object identification frame and at least one sub-target object identification frame, and a target object recognition result of the at least one target object identification frame and a sub-target object recognition result of the at least one sub-target object identification frame are determined.

4. The method according to claim 3, wherein The target object recognition frame is a human body recognition frame, and the sub-target object recognition frame includes at least one of the following: a face recognition frame and a head recognition frame.

5. The method according to claim 3, wherein Before grouping the at least one identification frame based on the positional relationship between the at least one identification frame to obtain the identification frames corresponding to the at least one target object, the method further includes: Iteratively performing a screening operation on a first screening object until the first screening object satisfies a first preset condition, wherein the initial first screening object is the at least one target object identification frame, and each iterative process includes the following screening operations: Obtaining a first analysis reference recognition frame with the highest probability of a target object recognition result from the first screening objects; Determine a first intersection-over-union ratio between the first analysis reference recognition frame and each of the other target object recognition frames in the first screening object; Remove the target object recognition frames whose first intersection-over-union ratio is greater than a first threshold from the other target object recognition frames; The first screening object is updated to the target object recognition frame retained in the other target object recognition frames.

6. The method according to claim 5, wherein Also includes: Iteratively perform a screening operation on the second screening object until the second screening object satisfies a second preset condition, wherein the initial second screening object is the at least one sub-target object identification frame, and each iterative process includes the following screening operations: Obtaining a second analysis reference recognition frame with the highest probability of sub-target object recognition result from the second screening objects; Determine a second intersection-over-union ratio between the second analysis reference recognition frame and each of the other sub-target object recognition frames in the second screening object; Remove the sub-target object identification frames whose second intersection-over-union ratio is greater than a second threshold from the other sub-target object identification frames; The second screening object is updated to the sub-target object identification frame retained in each of the other sub-target object identification frames.

7. The method according to any one of claims 1 to 6, characterized in that: The determining a target analysis object from the at least one target object based on the areas of the respective identification frames includes: For each identification frame corresponding to each target object in the at least one target object, perform the following operations: For each recognition frame corresponding to a target object, obtaining a candidate recognition frame with the largest area from among the recognition frames corresponding to the target object; From the obtained candidate recognition frames, select the candidate recognition frame with the largest area as the target recognition frame; The target object corresponding to the target identification frame in at least one target object is used as a target analysis object.

8. The method according to claim 7, wherein The determining of the overall recognition result of the target analysis object based on the partial recognition results of each recognition frame corresponding to the target analysis object includes: If the part incomplete recognition results of each recognition frame corresponding to the target analysis object all indicate that the parts are not incomplete, then the overall recognition result of the target analysis object is determined to be a complete object; otherwise, the overall recognition result of the target analysis object is determined to be an incomplete object.

9. The method according to claim 7, wherein The determining of the image recognition result of the image to be recognized based on the overall recognition result of the target analysis object includes: If the overall recognition result of the target analysis object is that the object is complete, determining that the image recognition result of the image to be recognized is a complete image; If the overall recognition result of the target analysis object is that the object is incomplete, then it is determined that the image recognition result of the image to be recognized is an incomplete image.

10. An image recognition device, characterized in that: include: a detection module configured to detect at least one target object in an image to be identified, obtain identification frames corresponding to the at least one target object, and determine partial incomplete identification results for each identification frame, wherein the at least one identification frame includes at least one target object identification frame and at least one sub-target object identification frame, wherein the sub-target object is a partial area of the target object, and the partial incomplete identification results for the at least one identification frame include a target object identification result and a sub-target object identification result, wherein the target object identification result is used to indicate whether the target object within the target object identification frame is complete, and the sub-target object identification result is used to indicate whether the sub-target object within the sub-target object identification frame is complete; a selection module, configured to determine a target analysis object from the at least one target object based on the areas of the respective recognition frames, wherein the target analysis object represents a main target object in the image to be recognized; A matching model is used to determine an overall recognition result of the target analysis object based on the partial incomplete recognition results of each recognition frame corresponding to the target analysis object; the overall recognition result includes whether the object is complete or incomplete; The discrimination module is used to determine the image recognition result of the image to be recognized based on the overall recognition result of the target analysis object.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that It stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of any one of the methods according to claims 1 to 9.

Images