An image recognition method and device based on artificial intelligence

Abstract

The application provides an image recognition method and device based on artificial intelligence; the method comprises: performing target detection processing on a to-be-recognized image to obtain a target object image from the to-be-recognized image; performing feature extraction processing based on the target object image to obtain corresponding image features; performing key point recognition processing based on the image features to obtain key points of the target object and corresponding positions; and determining the completeness of the target object in the to-be-recognized image based on the key points of the target object and the corresponding positions. Through the application, the target object can be accurately recognized in the image, and the incompleteness of the target object in the image can be flexibly determined.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to an image recognition method, apparatus, electronic device, and computer-readable storage medium based on artificial intelligence. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0003] Image recognition (AR) refers to the technology of using computers to process, analyze, and understand images in order to identify targets and objects of various patterns. With the development of artificial intelligence technology in recent years, image recognition technology has also been constantly innovating. Technologies such as facial recognition and body recognition are increasingly widely used in various fields. Accurately assessing the completeness of objects in the image to be recognized has become a significant challenge in image recognition processing.

[0004] When identifying whether an object in an image is complete, related technologies typically use simple classification models to make judgments. They rely on classification models to determine whether an image is complete, but rarely distinguish the objects to be identified and then assess the completeness of the target object. When identifying images in a targeted manner, this can lead to misjudgments of images that do not contain the target object, and it cannot accurately assess the completeness of the target object, thus affecting the accuracy and precision of the target object integrity assessment. Summary of the Invention

[0005] This application provides an image recognition method, apparatus, electronic device, and computer-readable storage medium based on artificial intelligence, which can accurately identify target objects and flexibly determine the degree of incompleteness of target objects in images.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides an image recognition method based on artificial intelligence, including:

[0008] Target detection processing is performed on the image to be identified in order to obtain the image of the target object from the image to be identified;

[0009] Based on the target object image, feature extraction processing is performed to obtain the corresponding image features;

[0010] Based on the image features, key point recognition processing is performed to obtain the key points of the target object and their corresponding positions.

[0011] Based on the key points and corresponding positions of the target object, the degree of completeness of the target object in the image to be identified is determined.

[0012] This application provides an image recognition device based on artificial intelligence, including:

[0013] The target detection module is used to perform target detection processing on the image to be identified, so as to obtain the target object image from the image to be identified;

[0014] The feature extraction module is used to perform feature extraction processing based on the target object image to obtain the corresponding image features;

[0015] The key point recognition module is used to perform key point recognition processing based on the image features to obtain the key points of the target object and their corresponding positions.

[0016] The integrity determination module is used to determine the degree of integrity of the target object in the image to be identified based on the key points of the target object and their corresponding positions.

[0017] In the above scheme, the target detection module is further used for:

[0018] The image to be recognized is processed for target detection to obtain a detection bounding box that includes the target object.

[0019] Based on the position of the detection box, the target object image is cropped from the image to be identified.

[0020] In the above scheme, the target detection module is further used for:

[0021] Extract the features of the image to be identified to obtain the corresponding feature map;

[0022] Multiple candidate bounding boxes are determined in the image to be identified;

[0023] The candidate bounding boxes are mapped onto the feature maps to obtain multiple corresponding candidate feature maps;

[0024] Max pooling is performed on the multiple candidate feature maps to obtain multiple candidate region maps of the same size;

[0025] The multiple candidate region maps are classified and candidate box position regression is performed to obtain a detection box including the target object.

[0026] In the above scheme, the feature extraction module is further used for:

[0027] The surrounding area of ​​the target object image is filled, and feature extraction processing is performed on the filled target object image to obtain the corresponding image features, or...

[0028] The cropped target object image is directly subjected to feature extraction processing to obtain the corresponding image features.

[0029] In the above scheme, when the image features are extracted from the padded target object image, the key point recognition module is further configured to:

[0030] The first keypoint detection model is invoked to perform the following processing:

[0031] The image features are mapped into a probability map with multiple channels, wherein the probability map of each channel corresponds to the probability distribution of a key point in a preset set of key points, and the probability distribution is used to characterize the probability that each pixel in the target object image belongs to the key point corresponding to the probability map;

[0032] For each probability map, the following processing is performed: the pixel with the highest probability in the probability map is identified as the key point corresponding to the probability map, and the position of the pixel with the highest probability is identified as the position of the key point corresponding to the probability map;

[0033] The key points and their corresponding positions identified from each probability map are combined to form the key point identification result of the target object;

[0034] The key point identification result includes multiple key points and their corresponding locations, and the multiple key points correspond one-to-one with all key points in the preset key point set.

[0035] In the above scheme, when the image features are extracted from the padded target object image, the key point recognition module is further configured to:

[0036] The second keypoint detection model is invoked to perform the following processing:

[0037] The image features are mapped to a probability map, wherein the probability map includes the probability of each pixel in the target object image corresponding to each key point in a preset set of key points;

[0038] Perform the following processing on each pixel in the target object image:

[0039] Determine the maximum probability among the probabilities of the pixel corresponding to each key point in the preset key point set;

[0040] When the maximum probability exceeds the probability threshold, the pixel is identified as the key point corresponding to the maximum probability, and the position of the pixel is identified as the position of the key point corresponding to the maximum probability.

[0041] The key points and their corresponding positions identified from the probability map are combined to form the key point identification result of the target object;

[0042] The key point identification result includes at least one key point and its corresponding position in the preset key point set.

[0043] In the above scheme, when the image features are extracted from a padded target object image, the integrity determination module is further configured to:

[0044] Perform the following processing for each key point of the target object:

[0045] When the location of the key point is within the surrounding area used for the filling process, the key point is determined to be a missing key point in the image to be identified.

[0046] Remove the missing key points from the identified key points of the target object to update the identified key points of the target object;

[0047] The ratio of the number of key points of the target object obtained after the update to the preset number of key points is used as the completeness of the target object in the image to be identified; wherein, the preset number of key points is the key point count of the preset key point set of the target object.

[0048] In the above scheme, when the image features are extracted directly from the target object image, the integrity discrimination module is further configured to:

[0049] The ratio of the number of key points of the target object obtained by the key point recognition process to the preset number of key points is used as the degree of completeness of the target object in the image to be recognized; wherein, the preset number of key points is the key point count of the preset key point set of the target object.

[0050] In the above scheme, before determining the ratio, the integrity discrimination module is further used to:

[0051] Occlusion recognition processing is performed on each key point of the target object to determine the occluded key points in the key points of the target object image;

[0052] The occluded key points are removed from the key points of the target object to update the key points of the target object. In the above scheme, the AI-based image recognition device further includes:

[0053] The processing module is configured to remove the image to be identified from the candidate cover image set when the image to be identified is a candidate cover image of a media account and the completeness of the image to be identified is lower than the completeness threshold; and to block the recommendation of the image to be identified or reduce the recommendation weight of the image to be identified when the image to be identified is carried in the information to be recommended and the completeness of the image to be identified is lower than the completeness threshold.

[0054] This application provides an electronic device, including:

[0055] Memory, used to store executable instructions;

[0056] A processor, when executing executable instructions stored in the memory, implements the method provided in the embodiments of this application.

[0057] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the artificial intelligence-based image recognition method provided in this application.

[0058] The embodiments of this application have the following beneficial effects:

[0059] Explicit target detection was performed on the image to be recognized to perform key point recognition processing on the target object image. That is, when it is determined that the target object is included, subsequent key point recognition is performed to determine the degree of incompleteness of the target object. This eliminates the erroneous judgment caused by images that do not contain the target object, and accurately and efficiently obtains the degree of completeness of the target object. Attached Figure Description

[0060] Figure 1 This is an optional structural diagram of the image recognition system architecture based on artificial intelligence provided in the embodiments of this application;

[0061] Figure 2 This is an optional structural schematic diagram of the image recognition device based on artificial intelligence provided in the embodiments of this application;

[0062] Figure 3A This is an optional flowchart illustrating an image recognition method based on artificial intelligence provided in an embodiment of this application;

[0063] Figure 3B This is an optional flowchart illustrating an image recognition method based on artificial intelligence provided in an embodiment of this application;

[0064] Figure 4 This is an optional structural diagram of the target detection model provided in the embodiments of this application;

[0065] Figure 5This is an optional structural diagram of the target detection model provided in the embodiments of this application;

[0066] Figure 6 This is an optional schematic diagram of the principle of the method for obtaining candidate boxes provided in the embodiments of this application;

[0067] Figure 7 This is an optional schematic diagram of the filling process method provided in the embodiments of this application;

[0068] Figure 8 This is an optional structural diagram of the key point recognition model provided in the embodiments of this application;

[0069] Figure 9 This is an optional schematic diagram of the feature extraction processing method provided in the embodiments of this application;

[0070] Figure 10 This is an optional schematic diagram of a preset set of key points provided in the embodiments of this application;

[0071] Figure 11 This is an optional flowchart illustrating an image recognition method based on artificial intelligence provided in an embodiment of this application;

[0072] Figure 12 This is a schematic diagram of an optional process for applying the image recognition method provided in this application to the selection of cover images for media accounts;

[0073] Figure 13 This is an optional flowchart illustrating the application of the image recognition method provided in this application to a recommendation system. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0075] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0076] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0078] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0079] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0080] 1) Object detection: also known as object extraction, is an image segmentation technique based on the geometric and statistical features of objects. It combines object segmentation and recognition into one, that is, finding all objects of interest in an image, including two sub-tasks: object localization and object classification, while determining the category and location of the objects.

[0081] 2) Key point: In image processing, a feature description is an abstract description of a fixed region or spatial physical relationship. It describes the combination or contextual relationship within a certain domain, including but not limited to point information, and further represents the relationship between a location or context and its surrounding neighborhood.

[0082] 3) Downsampling: This also means reducing the image size. For example, for an image of size M*N, downsampling it by a factor of s results in a resolution image of size (M / s)*(N / s), where s is the common divisor of M and N. For a matrix-like image, the image within an s*s window of the original image is transformed into a single pixel, and the value of this pixel is the mean or maximum value of all pixels within that window.

[0083] 4) Upsampling: also known as upsampling the image, usually uses interpolation methods to insert new elements between pixels based on the original image pixels using a suitable interpolation algorithm.

[0084] 5) Non-maximum Suppression (NMS): Suppresses elements that are not maxima; it can be understood as a local maximum search. It is used in object detection to extract the highest-scoring window (detection box). For example, in object detection, after feature extraction and classification by a classifier, a score is obtained for each sliding window. Since sliding windows can lead to complete overlap or significant intersection between windows, NMS is used to select those windows with the highest scores in their neighborhoods (those with the highest probability of being the target object) and suppress those windows with low scores.

[0085] Existing technologies for incompleteness identification in image recognition processing simply use classification models to make judgments. If the image does not contain the target object to be identified, it will be identified as complete by default. Classification-based methods lack an explicit process for identifying the target object, leading to incorrect incompleteness classifications for images that do not contain the target object. Furthermore, existing classification models can only output two final results: complete or incomplete target object, failing to determine the degree of incompleteness and thus struggling to meet the needs of different business scenarios regarding target object incompleteness identification.

[0086] Based on this, embodiments of this application provide an image recognition method, apparatus, electronic device, and computer-readable storage medium based on artificial intelligence, which can reduce misjudgments of the incompleteness of target objects in the image to be recognized and accurately and flexibly obtain the degree of incompleteness of target objects in the image to be recognized.

[0087] First, the image recognition system based on artificial intelligence provided in the embodiments of this application will be described, see [link to relevant documentation]. Figure 1 , Figure 1 This is an optional architecture diagram of an AI-based image recognition system 100 provided in this application embodiment. To support an AI-based image recognition application, the terminal 400 connects to the server 200 via the network 300.

[0088] The AI-based image recognition system provided in this application can be applied to media account cover image selection. Specifically, terminal 400 acquires candidate cover images as images to be recognized, generates an image recognition request based on the images to be recognized, and terminal 400 or server 200 responds to the image recognition request by implementing the AI-based image recognition method provided in this application to obtain the degree of incompleteness of the target object in the image to be recognized and determine the completeness result of the target object. Terminal 400 processes the candidate cover images according to the completeness recognition result of the target object. For example, if the target object is complete, the candidate cover image is displayed as the cover image of the media account on the user interface; if the target object is incomplete, the current candidate cover image is discarded.

[0089] The AI-based image recognition system provided in this application can also be applied to online recommendation systems. Specifically, terminal 400 acquires an image to be recommended, uses it as an image to be recognized, generates an image recognition request based on the image to be recognized, and terminal 400 or server 200 responds to the image recognition request by implementing the AI-based image recognition method provided in this application to obtain the degree of incompleteness of the target object in the image to be recognized, determine the completeness result of the target object, and terminal 400 processes the image to be recommended according to the completeness recognition result of the target object. For example, if the target object is complete, the image to be recommended is displayed on the user interface; if the target object is incomplete, the current image to be recommended is not recommended or its recommendation priority is reduced.

[0090] In some embodiments, the implementation of the AI-based image recognition method provided in this application in response to an image recognition request can be carried out interactively by a terminal and a server, as specifically described below: Terminal 400 is used to send an image recognition request to server 200, where the image recognition request includes an image to be recognized; Server 200 is used to receive the image recognition request, respond to and parse the image recognition request to obtain the image to be recognized, respond to the request, perform target detection processing on the image to be recognized to obtain a target object image from the image to be recognized; perform feature extraction processing based on the target object image to obtain corresponding image features; perform key point recognition processing based on the image features to obtain the key points of the target object and their corresponding positions; determine the completeness of the target object in the image to be recognized based on the key points and their corresponding positions; determine the completeness of the target object based on the completeness of the target object in the image to be recognized, and return the image recognition result to terminal 400 as the image recognition result.

[0091] In other embodiments, the implementation of the AI-based image recognition method provided in this application in response to an image recognition request can be carried out independently by the terminal 400. Specifically, the terminal 400 can respond to an image recognition request and implement the AI-based image recognition method provided in this application based on the image to be recognized to obtain the degree of completeness of the target object in the image to be recognized and determine the completeness of the target object.

[0092] In some embodiments, terminal 400 may be a laptop computer, tablet computer, desktop computer, smartphone, dedicated messaging device, portable gaming device, smart speaker, smartwatch, etc., but is not limited thereto.

[0093] Server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. Terminal 400 and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0094] The electronic device for implementing the above-described artificial intelligence-based image recognition method, as provided in the embodiments of this application, will be described next. In practical applications, the electronic device can be implemented as follows: Figure 1 Terminal 400 or server 200 in the middle, with electronic devices as Figure 1 Taking server 200 as an example, an electronic device implementing the artificial intelligence-based image recognition method of this application will be described. See also Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2 The server 200 shown includes at least one processor 210, memory 250, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0095] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0096] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0097] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0098] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0099] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0100] The network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0101] In some embodiments, the AI-based image recognition device provided in this application can be implemented in software. Figure 2 An AI-based image recognition device 255, stored in memory 250, is shown. This device can be software in the form of programs and plugins, and includes the following software modules: an object detection module 2551, a feature extraction module 2552, a key point recognition module 2553, and an integrity determination module 2554. These modules are logically connected and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0102] In other embodiments, the AI-based image recognition device provided in this application can be implemented in hardware. As an example, the AI-based image recognition device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the AI-based image recognition method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0103] The image recognition method for artificial intelligence provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0104] See Figure 3A , Figure 3A This is an optional flowchart illustrating an artificial intelligence-based image recognition method provided in an embodiment of this application, which will be combined with... Figure 3A Steps 101-104 shown are explained.

[0105] Step 101: Perform target detection processing on the image to be recognized to obtain the target object image from the image to be recognized.

[0106] Here, the image to be identified can be acquired in any way. For example, it can be acquired by taking a picture with a camera or other tool on a terminal device, by cropping from a video frame, or by acquiring it from an image dataset. This application embodiment does not limit this.

[0107] It should be noted that the target object here can be any object to be identified, such as animals or people. For animals, the object detection process can identify the detection box of a certain animal in the image to be identified through the object detection model and obtain the corresponding position of the animal. For people, the object detection process can identify the detection box containing different human bodies (or faces) in the image to be identified through the object detection model and obtain the position of the corresponding human body (or face).

[0108] In some embodiments, see Figure 3B , Figure 3B This is an optional flowchart illustrating an image recognition method based on artificial intelligence provided in an embodiment of this application. Based on Figure 3AThe target detection processing of the image to be identified in step 101 to obtain the target object image from the image to be identified can be specifically implemented through the following steps 1011-1012.

[0109] Step 1011: Perform target detection processing on the image to be recognized to obtain a detection box including the target object.

[0110] In some embodiments, the target detection processing of the image to be recognized in step 1011 to obtain a detection box including the target object is specifically implemented through the following technical solution: extracting features of the image to be recognized to obtain a corresponding feature map; determining multiple candidate boxes in the image to be recognized; mapping the candidate boxes to the feature map to obtain multiple candidate feature maps; performing max pooling processing on the multiple candidate feature maps to obtain multiple candidate region maps of the same size; performing classification processing and candidate box position regression processing on the multiple candidate region maps to obtain a detection box including the target object.

[0111] In some embodiments, the above-described object detection processing can be implemented by an object detection model, which may be a Region Convolutional Neural Network (RCNN), a Faster Region Convolutional Neural Network (Faster-RCNN), an EfficientDet network model, etc.

[0112] In some embodiments, the object detection process in step 1011 is illustrated using the Faster-RCNN model as an example. See [link to documentation]. Figure 4 , Figure 4 This is an optional structural diagram of the target detection model provided in the embodiments of this application. Here, the target detection model is the Faster-RCNN model, which is used to implement the method provided in step 1011 to perform target detection processing on the image to be recognized and obtain a detection box including the target object.

[0113] Specifically, in Figure 4 In Faster-RCNN, the model includes convolutional layers (Conv layers), a region proposal network (RPN), a region of interest (ROI) pooling layer, and a classification layer.

[0114] In some embodiments, based on Figure 4The Faster-RCNN model shown extracts features from the image to be recognized using convolutional layers to obtain corresponding feature maps. Based on the RPN network, multiple candidate boxes are determined in the image to be recognized. These candidate boxes are mapped onto the feature maps to obtain multiple candidate feature maps. A region-of-interest (ROI) pooling layer is then used to perform max pooling on these candidate feature maps, resulting in multiple candidate region maps of the same size. A classification layer is then used to classify these candidate region maps to obtain the image category results corresponding to the multiple candidate boxes. Finally, by combining the results of candidate box position regression processing on the multiple candidate region maps, target detection boxes containing different categories of objects are obtained in the image to be recognized.

[0115] Here, based on business requirements under specific conditions, a detection box including the target object can be selected in the detection box of the image to be recognized according to the object category. Continuing from the above embodiments, a detection box containing the object category of animal, a detection box containing the object category of human face, or a detection box containing the object category of human body, etc., can be selected in the detection box.

[0116] In actual implementation, see Figure 5 , Figure 5 This is an optional structural diagram of the object detection model provided in the embodiments of this application. Here, the object detection model is the Faster-RCNN model. Figure 4 The model structure of the Faster-RCNN model shown can be specifically represented as consisting of... Figure 5 The model structure of the Faster-RCNN model is shown. Figure 5 In Faster-RCNN, the network specifically includes a feature extraction network, an RPN network, and an RCNN network. The feature extraction network includes a certain number of convolutional (conv) layers, non-linear transformation (reLU) layers, and pooling layers. The RPN network includes a certain number of convolutional layers and non-linear transformation layers. The RCNN network includes region-of-interest pooling layers and two fully connected layers.

[0117] In some embodiments, based on Figure 4 The convolutional layers in the image extract features from the image to be recognized to obtain the corresponding feature map. Figure 5The feature extraction network is implemented in the following way. Here, the feature extraction network can be a VGG16 network or a ZF network. For example, taking the VGG16 network as an example, the feature extraction network can contain 13 convolutional layers, 13 nonlinear transformation layers, and 4 pooling layers. Specifically, extracting features from the image to be recognized through the feature extraction network to obtain the corresponding feature map can be implemented as follows: convolutional processing of the image is performed using convolutional layers and nonlinear transformation layers to extract the feature information of the image, and downsampling operation of the image is performed through pooling layers to pool the features. After multiple convolution, nonlinear transformation, and pooling operations, the feature map corresponding to the image to be recognized is obtained.

[0118] Here, the kernel size of all convolutional layers is set to 3*3, with 1-pixel edge padding. The kernel size of all downsampling layers is set to 2*2. At this point, the input image size is M*N. After 1-pixel edge padding, the image size becomes (M+2)*(N+2). After another 3*3 convolution, the output feature map size is the same as the input image size. The downsampling layer kernel size is 2*2 with a stride of 2. When the input matrix is ​​M*N, the output matrix becomes (M / 2)*(N / 2). Through convolution and pooling operations, the dimensionality of the original input image is reduced to half. After four pooling layers, the feature map output by the feature extraction network is 1 / 256 the size of the input image, which is used to quickly map the features back to the original image to be recognized.

[0119] In some embodiments, based on Figure 4 The RPN network in the image determines multiple candidate boxes in the image to be recognized. Figure 5 The implementation of the RPN network in [the document / section]. Here, the backbone of the RPN network can be a VGG16 network or a ZF network. For example, taking a VGG16 network as an example, the RPN network can include a certain number of convolutional layers and nonlinear transformation layers. Specifically, in [the document / section]... Figure 5 In the RPN, for the feature map output by the feature extraction network, a 3*3 convolutional kernel rpn_conv with 1-pixel edge padding is used for one convolution, followed by non-linear transformation. During the convolution process, multiple anchors in the image to be recognized are identified, and the features of these anchors are obtained. Based on the features of these anchors, they are fed into two parallel processing branches. The first branch uses a 1*1 convolutional kernel to process the anchors and then performs classification processing. The second branch uses a 1*1 convolutional kernel to process the anchors and then performs candidate box regression processing. Finally, multiple candidate boxes are obtained based on the classification results and the candidate box regression results.

[0120] In some embodiments, classification of multiple anchors can be implemented as follows: For each anchor, the anchor's features are reshaped, and a softmax function is used for probability mapping, mapping the features of multiple anchors to foreground / background probabilities (here, the target object to be identified is defined as the foreground, and the rest is defined as the background). Then, the dimensions are changed, and the classification result for each anchor corresponding to foreground / background is output. It should be noted that since softmax classifies by channels, the data is reshaped to create a separate dimension before reshaping it to return to the original data structure.

[0121] In some embodiments, anchors belonging to the foreground are selected from multiple anchors. Their position coordinates are corrected using boundary regression, and they are sorted according to the softmax output score. The top N anchors are extracted, and after non-maximum suppression (NMS), redundant anchors are removed. Then, the top M anchors with the highest output scores are selected again to determine multiple candidate boxes. Here, the NMS filtering of redundant candidate boxes can be implemented as follows: Candidate boxes for each category are sorted in descending order of classification confidence; within each category, the candidate box with the highest confidence is selected, and the intersection-union ratio (IU) of the highest-confidence candidate box and the remaining candidate boxes is calculated. Candidate boxes with an IU greater than a threshold are deleted.

[0122] For example, see Figure 6 , Figure 6 This is an optional schematic diagram illustrating the principle of the method for obtaining candidate boxes provided in an embodiment of this application. Figure 6 The method for obtaining candidate boxes is implemented by an RPN network; Figure 5 The provided method identifies multiple anchors in an image to be recognized and obtains features from these anchors, along with a two-branch processing method. Specifically, this can be achieved through... Figure 6The specific steps in the schematic diagram are explained below: When performing convolution processing on the feature map, k anchors are generated in the image to be recognized based on the center point of each sliding convolution kernel window. Here, three aspect ratios (1:1, 1:2, 2:1) and three area scales (128*128, 256*256, 512*512) are typically used as constraints for anchor selection, resulting in 9 anchors. The features of these 9 anchors after convolution processing are then obtained. Based on the features of each anchor, foreground / background classification is performed for each anchor, and candidate box position regression is performed (here, candidate box position regression is performed on the candidate boxes using pre-labeled target boxes, obtaining the horizontal and vertical coordinates of the center point, and four offsets: length and width). Taking the VGG network as an example, each point in the feature map is 512-dimensional. After each anchor is mapped onto the feature map, the two branches process it in parallel, transforming it into a 2*k-dimensional feature map (representing the classification result of foreground / background probability) and a 4*k-dimensional feature map (representing the position regression result, i.e., the four offsets).

[0123] In some embodiments, based on Figure 4 The region-of-interest pooling layer in the code performs max pooling on multiple candidate feature maps, resulting in multiple candidate region maps of the same size. Figure 5 The region of interest pooling layer is implemented.

[0124] For example, the ROI polling layer divides the input candidate feature maps of different proportions and scales into 7 equal parts in both the horizontal and vertical directions, resulting in 49 regions. For each region, the maximum value downsampling process is performed to obtain multiple candidate region maps of the same size.

[0125] In some embodiments, based on Figure 4 The classification layer in the code implements a method to classify multiple candidate region maps to obtain multiple candidate boxes. Figure 5 The fully connected layer implementation in the first branch, and, Figure 5 The fully connected layer of the second branch shown is used for candidate box position regression processing.

[0126] For example, the following processing is performed on each candidate region map: classification processing is performed through the fully connected layer of the first branch: each candidate region map is mapped to a probability matrix belonging to each category, and the corresponding object category in the image to be identified is determined; candidate box regression processing is performed using the fully connected layer of the parallel second branch to obtain the position offset to correct the position of the candidate box; after classification processing and candidate box regression processing are performed on all candidate region maps, the corresponding candidate boxes are selected to obtain target detection boxes containing different categories of objects in the image to be identified.

[0127] Step 1012: Based on the position of the detection box, crop the target object image from the image to be identified.

[0128] In practice, the image to be identified is cropped based on the position of the detection box in the image to be identified, and the target object image with the same size as the detection box is extracted.

[0129] For example, based on the location of the detection box containing the human body, the image of the human body is cropped out at the corresponding location in the image to be identified. It should be noted that, generally speaking, the detection box obtained after object detection processing contains a single human body.

[0130] Here, the target object is identified in the image to be recognized. For example, the human body is identified and screened in the image through object detection. This avoids the misjudgment of images that do not include human bodies when the existing classification model makes incomplete judgments on human bodies in the image. When applied to specific image recognition business processing, it improves the accuracy of object integrity judgment.

[0131] Step 102: Perform feature extraction processing based on the target object image to obtain the corresponding image features.

[0132] In some embodiments, feature extraction processing based on the target object image to obtain the corresponding image features is specifically achieved through the following technical solution: filling the surrounding area of ​​the target object image, and performing feature extraction processing on the filled target object image to obtain the corresponding image features.

[0133] In practice, a ring of uniform pixel values ​​is filled around the target object image to obtain a target object image of a preset size after filling, which is then used for feature extraction and key point recognition.

[0134] For example, see Figure 7 , Figure 7 This is an optional schematic diagram of the filling process method provided in the embodiments of this application. Figure 7 Region 71 is the target object image cropped from the image to be identified (hereinafter referred to as the original target image), and region 72 is the surrounding area filled with uniform pixel values. Here, in order to adapt to different detection box sizes, the original target image is adjusted to a preset size, and the area outside the original target image within the preset size is filled with uniform pixel values ​​to obtain the filled target object image.

[0135] It should be noted that the uniform pixel value is an empirical pixel value used to characterize the surrounding area as an area outside the original target image, belonging to the background or occluded area compared to the original target image. Here, the uniform pixel value can be a pre-set specific pixel value, such as the average pixel value calculated in a large image recognition database (ImageNet), etc. This application embodiment does not make specific limitations on the setting of the uniform pixel value. In some embodiments, the above feature extraction processing can be implemented by a key point recognition model, which can be a Cascaded Pose Regression (CPR) model, a high-resolution network model (HRNet), etc.

[0136] In some embodiments, padded target object images can be stored in an image database or image sample set to be used as training sample images for training a keypoint recognition model. Here, the keypoints of the target object are marked in the padded target object image, and the padded target object image and the marked keypoints of the target object are used as training sample images so that the keypoint recognition model can recognize the keypoints after extracting image features from the surrounding area.

[0137] In some embodiments, the HRnet model is used as an example. The HRnet model includes two parts: a feature extraction part and a key point recognition part. The feature extraction process based on the target object image in step 102 to obtain the corresponding image features can be achieved through the feature extraction part of HRnet.

[0138] Here, typical keypoint recognition models first downsample the input image to reduce its resolution, and then upsample it to increase the resolution, resulting in the final feature extraction result. The method of downsampling to retain strong image features and then upsampling to restore image resolution will lose some spatial information. When the feature extraction result is used for keypoint recognition, it will produce a certain quantization error. HRnet adopts a parallel network structure design, always keeping one branch for high-resolution feature extraction, thereby protecting rich spatial information and reducing the generation of quantization error.

[0139] See Figure 8 , Figure 8 This is an optional structural diagram of the key point recognition model provided in the embodiments of this application. Here, the key point recognition model is the HRNet model. Figure 8In the diagram, the horizontal direction represents the network depth, and the vertical direction represents the network scale. Depths 1 to 15 represent the network structure of the feature extraction part of HRNet. The feature extraction part can be divided into multiple stages (e.g., depths 1 to 3, 4 to 6, 7 to 10, and 11 to 14 are different stages).

[0140] In practice, before each stage, a smaller resolution feature map (also smaller in scale) is added. Feature maps of different scales are obtained through interpolation upsampling and convolution downsampling, respectively. Feature maps of the same scale are then fused to ensure that the initial feature map incorporates features from different scales in the previous stage. During each stage, a Residual Neural Network (ResNet) is used to perform deep learning on the feature maps at each scale. After the final stage (depths 11 to 14), four feature maps of different scales are obtained. Based on these four feature maps, the final output feature map at depth 15 is obtained. For details, see [link to details]. Figure 9 , Figure 9 This is an optional schematic diagram of the feature extraction processing method provided in the embodiments of this application. Figure 9 In this process, feature maps of the same size as the largest scale feature map (which always maintains high resolution) are obtained by interpolating and upsampling all the small-scale feature maps. For example, feature maps 902, 903, and 904 are interpolated and upsampled respectively to obtain feature maps 905, 906, and 907 of the same size as feature map 901. Feature maps 901, 905, 906, and 907 are then fused together to obtain the final output feature map at a depth of 15. The feature map is used to characterize the feature information of the target object image.

[0141] Here, feature extraction of the target object image after filling is beneficial to detect all theoretically obtainable key points in the target object image, so as to quantify the degree of incompleteness of the target object based on the relationship between the position of the key points and the surrounding area of ​​the filling process.

[0142] In other embodiments, the feature extraction process based on the target object image to obtain the corresponding image features is further implemented through the following technical solution: directly performing feature extraction processing on the cropped target object image to obtain the corresponding image features.

[0143] In actual implementation, the feature extraction part of HRnet directly performs feature extraction processing on the cropped target object image to obtain the corresponding image features. Here, the feature extraction processing can refer to the feature extraction processing method of the embodiment of this application, which will not be described in detail here.

[0144] Here, feature extraction from the directly cropped target object image helps to detect the key points that actually exist in the target object image, intuitively see the missing key points of the target object, and easily obtain the degree of incompleteness of the target key points.

[0145] Step 103: Perform key point recognition processing based on image features to obtain the key points of the target object and their corresponding locations.

[0146] In some embodiments, when image features are extracted from a padded target object image, keypoint recognition processing based on image features is performed to obtain the keypoints of the target object and their corresponding positions. This is specifically achieved through the following technical solution: A first keypoint detection model is invoked to perform the following processing: Image features are mapped to a probability map with multiple channels, where each channel's probability map corresponds to the probability distribution of a keypoint in a preset keypoint set. The probability distribution characterizes the probability that each pixel in the target object image belongs to the keypoint corresponding to the probability map. For each probability map, the following processing is performed: The pixel with the highest probability in the probability map is identified as the keypoint corresponding to the probability map, and the position of the pixel with the highest probability is identified as the position of the keypoint corresponding to the probability map. The keypoints and their corresponding positions identified from each probability map are combined to form the keypoint recognition result of the target object. The keypoint recognition result includes multiple keypoints and their corresponding positions, and these multiple keypoints correspond one-to-one with all keypoints in the preset keypoint set.

[0147] Here, the first keypoint detection model can be the keypoint recognition model mentioned above, such as the HRNet model. Continuing from the previous point, the feature extraction part of the first keypoint detection model is called to perform feature extraction processing on the padded target image object, obtaining the corresponding feature map. Based on the feature map, the keypoint recognition part of the first keypoint detection model is called to perform keypoint recognition processing.

[0148] In actual implementation, the key point recognition part adds a convolutional layer after the feature map to map the image features into a probability map of multiple channels. Here, the number of channels is set to the number of key points in the preset key point set so that the probability map of each channel corresponds to the probability distribution of a key point in the preset key point set.

[0149] In some embodiments, in each probability map corresponding to different channels, the pixel with the highest probability in the probability map can be identified as the key point corresponding to the probability map, and the position of the pixel with the highest probability can be identified as the position of the key point corresponding to the probability map. Specifically, the coordinates of the key point can be determined by the following formula (1):

[0150] (1)

[0151] Where Pi is the coordinate of the key point, Heatmapi is the predicted heatmap corresponding to the i-th channel, i.e., the probability map, and i is a positive integer, i≥1.

[0152] In other embodiments, each probability map corresponding to different channels can be a truth probability map, and pixels with a probability value of 1 in the truth probability map can be identified as key points corresponding to the truth probability map.

[0153] The training process of the first keypoint detection model is described below. Before calling the first keypoint model, a first training sample set is also obtained to train the first keypoint detection model. Here, the first training sample set includes: occluded image samples and unoccluded image samples, and the occluded and unoccluded image samples carry keypoint location markers. In the occluded image samples, some keypoints of the target object are occluded by the background, while the keypoints of the object in the unoccluded image samples are not occluded. Here, the partial occlusion of keypoints by the background can be any occlusion situation. For example, it can be the filling process according to the embodiments of this application, where uniform pixel values ​​are filled at the corresponding keypoint positions to serve as a background area, occluding the corresponding keypoints, and marking the positions of the keypoints in the occluded area. The first keypoint detection model is trained through the above training sample set so that the keypoint recognition model can successfully recognize keypoints in the occluded area, forming a keypoint recognition result of the target object. The keypoint recognition result includes multiple keypoints corresponding to all keypoints in the preset keypoint set corresponding to the target object, as well as the positions of the keypoints.

[0154] Here, the preset keypoint set is a pre-defined set of points representing key information of the target object. For example, when the target object is the human body, the preset keypoint set for each human body is defined to include 17 key points. See [link to relevant documentation]. Figure 10 , Figure 10 This is an optional schematic diagram of a preset key point set provided in the embodiments of this application. Here, the preset key point set consists of 17 key points of the human body, namely the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

[0155] Here, the first key point detection model is used to identify key points in the target object image after filling. This ensures that no key point information is lost, so that the missing key point information of the target object can be determined based on the positions of all identified key points, thus accurately obtaining the completeness of the target object.

[0156] In other embodiments, when the image features are directly extracted from the target object image, keypoint recognition processing based on the image features is performed to obtain the keypoints of the target object and their corresponding positions. This is specifically achieved through the following technical solution: A second keypoint detection model is invoked to perform the following processing: the image features are mapped to a probability map, wherein the probability map includes the probability of each pixel in the target object image corresponding to each keypoint in a preset keypoint set; for each pixel in the target object image, the following processing is performed: the maximum probability among the probabilities of the pixel corresponding to each keypoint in the preset keypoint set is determined; when the maximum probability exceeds a probability threshold, the pixel is identified as the keypoint corresponding to the maximum probability, and the position of the pixel is identified as the position of the keypoint corresponding to the maximum probability; the keypoints identified from the probability map and their corresponding positions are combined to form the keypoint recognition result of the target object; wherein the keypoint recognition result includes at least one keypoint and its corresponding position in the preset keypoint set.

[0157] Here, the second keypoint detection model can also be the keypoint recognition model mentioned above, such as the HRNet model. Continuing from the previous point, the feature extraction part of the second keypoint detection model is called to perform feature extraction processing on the directly cropped target object image to obtain the corresponding feature map. Based on the feature map, the keypoint recognition part of the second keypoint detection model is called to perform keypoint recognition processing.

[0158] In practical implementation, the keypoint recognition part adds a fully connected layer after the feature map to flatten the feature map, mapping the image features into a probability map. This probability map can be a probability matrix, representing the probability of each pixel corresponding to each keypoint in a preset set of keypoints. Here, the mapping can be a multi-class mapping, where the probability of each probability point represents the probability of belonging to each of multiple keypoint categories. For each pixel, the maximum probability of each pixel is determined and compared with a preset probability threshold. When the maximum probability exceeds the threshold, the keypoint category corresponding to the maximum probability is taken as the classification result for that pixel. If the maximum probability does not exceed the threshold, the current pixel recognition fails, indicating that the current pixel does not belong to any type of keypoint.

[0159] Here, the probability threshold represents the threshold value of the probability. The probability threshold can be set to 80%. For the maximum probability exceeding the probability threshold, the category corresponding to the maximum probability is usually the category to which the current pixel belongs or is most likely to belong to a certain key point.

[0160] The training process of the second keypoint detection model is described below. Before calling the second keypoint model, a second training sample set is also obtained to train the first keypoint detection model. Here, the second training sample set is used to contain image sample sets of the target object for keypoint recognition of the target object. The second training sample set includes: incomplete object image samples and complete object image samples. The complete object image samples contain all keypoints in the preset keypoint set corresponding to the target object, and the incomplete object image samples contain at least one keypoint in the preset keypoint set corresponding to the target object. The second keypoint detection model is trained using the above training sample set so that the second keypoint detection model can directly perform feature extraction and keypoint recognition processing on the cropped target object image to form the keypoint recognition result of the target object. The keypoint recognition result includes at least one keypoint in the preset keypoint set corresponding to the target object and the position of the keypoint.

[0161] Here, a second keypoint detection model is used to identify keypoints in the target object image obtained after direct cropping. The identified keypoints are then compared with a preset set of keypoints for the target object, allowing for a quick and convenient determination of the missing keypoint information. This, in turn, determines the completeness of the target object.

[0162] Step 104: Determine the completeness of the target object in the image to be recognized based on the key points of the target object and its corresponding position.

[0163] In some embodiments, when image features are extracted from a padded target object image, the completeness of the target object in the image to be recognized is determined based on the key points of the target object and their corresponding positions through the following technical solution: For each key point of the target object, the following processing is performed: when the key point is located in the surrounding area used for padded processing, the key point is determined to be a missing key point in the image to be recognized; the missing key points are removed from the identified key points of the target object to update the identified key points of the target object; the ratio of the updated number of identified key points of the target object to a preset number of key points is used as the completeness of the target object in the image to be recognized; wherein, the preset number of key points is the key point count of a preset set of key points of the target object.

[0164] In practical implementation, continuing from the previous point, when image features are extracted from a padded target object image, keypoint recognition is used to obtain the keypoint recognition results of the target object. The keypoint recognition results include multiple keypoints corresponding to all keypoints in a preset keypoint set corresponding to the target object, as well as the positions of these keypoints. For each identified keypoint, based on its coordinates in the image to be recognized, it is determined whether the current keypoint is located in the padded surrounding area or the original target image area. When the detected keypoint is located in... Figure 7 When the surrounding area 72 is shown, the target object obtained by target detection does not actually have the key information of this key point. Based on this, it is determined that the current key point is the missing key point of the target object, and the completeness of the target object in the image to be recognized is judged by comparing the number of missing key points with the preset key point set of the target object.

[0165] For example, when the target object is a human body, 17 key points and their corresponding location information are obtained through key point recognition. The relationship between these key points and the surrounding area is determined, and it is found that the key points for the left and right knees and ankles are located in the surrounding area. Therefore, it can be inferred that the target human body is missing at least the area below the left and right knees. To quantify the degree of incompleteness of the target human body, the missing key points (left and right knees and ankles) are removed from the identified key points, and the target key points are updated (resulting in 13 remaining key points). The ratio of the updated number of key points (13 key points) to the preset number of key points (17 key points) is used as the degree of completeness of the target object in the image to be recognized.

[0166] It should be noted that the key point in the embodiments of this application can be a single pixel or a set of multiple pixels. For example, the key point representing the nose can be composed of 7 pixels that fit the nose contour. In the examples of this application, the description of the key point count of the preset key point number as the preset key point set of the target object ignores the point set situation of each key point. In actual implementation, the situation that each key point is composed of a point set can be considered to enrich the count of key points in the preset key point set to optimize the ratio and more fully reflect the degree of completeness of the target object judged based on key point information. This embodiment of the application will not describe this further.

[0167] In some embodiments, a completeness threshold can be preset, which characterizes whether the target object contained in the image to be identified is complete. The completeness threshold is compared with the aforementioned completeness level; if the completeness level exceeds the completeness threshold, the current target object is determined to be a complete object. In actual implementation, to adapt to different business needs, the threshold can be flexibly set to determine the completeness of the target object.

[0168] Here, key point recognition is performed on the filled target object image, which can locate all theoretically obtainable key points in the image, determine the positional relationship between key points and the surrounding area of ​​the filled process and the original target image area, and clearly obtain the missing key point information in the image, so as to quickly determine the degree of incompleteness of the target object.

[0169] In other embodiments, when the image features are extracted from the image of the direct target object, the completeness of the target object in the image to be identified is determined based on the key points of the target object and their corresponding positions by the following technical solution: the ratio of the number of key points of the target object obtained by key point recognition processing to the preset number of key points is used as the completeness of the target object in the image to be identified; wherein, the preset number of key points is the key point count of the preset key point set of the target object.

[0170] In practical implementation, continuing from the previous point, when image features are directly extracted from the target object image, keypoint recognition processing is performed based on the image features to obtain the keypoint recognition result of the target object. The keypoint recognition result includes at least one keypoint from a preset keypoint set and its corresponding position. Here, the at least one keypoint obtained directly from the cropped target object image in the image to be recognized may be all keypoints in the preset keypoint set, or it may be some keypoints. When all keypoints in the preset keypoint set are recognized, the target object is generally considered complete without considering occlusion. When some keypoints are recognized, by comparing them with the preset keypoint set, it is determined which keypoint information is missing, thus obtaining the missing keypoint information of the target object and consequently, the completeness of the target object in the image to be recognized.

[0171] For example, when the target object is a human body, keypoint recognition yields nine keypoints (nose, left eye, left ear, left shoulder, left elbow, left wrist, left hip, left knee, and left ankle) along with their corresponding location information. Comparing these with the preset 17 keypoints for the target human body, it can be inferred that the target human body is missing at least its right side. To quantify the degree of incompleteness of the target human body, the ratio of the number of identified keypoints (9 keypoints) to the preset number of keypoints (17 keypoints) is used as the degree of completeness of the target object in the image to be recognized.

[0172] Here, key point recognition is directly performed on the target object image cropped from the image to be recognized to determine the completeness of the target object. The actual key information of the target object can be obtained simply and conveniently. By comparing the recognized key information with the theoretically obtained key information, the missing information of key points of the target object can be quickly obtained to determine the completeness of the target object.

[0173] In some other embodiments, before determining the ratio of the key points of the target object to the preset number of key points in step 104, the following processing may be performed: occlusion recognition processing is performed on each key point of the target object to determine the occluded key points in the key points of the target object image; the occluded key points are removed from the key points of the target object to update the key points of the target object.

[0174] Here, taking into full account that the key point recognition processing in this application embodiment shields the target object from situations such as object occlusion or object overlap, the detected key points are subjected to occlusion recognition processing before determining the ratio of the number of key points of the target object to the preset number of key points.

[0175] In some embodiments, occlusion recognition processing for each key point of the target object can be implemented through the following specific scheme: For each key point, perform binary classification recognition on the pixels representing key point information (binary classification means distinguishing between background category and foreground category, with background category being the occlusion category), convert the key point information into a feature vector, and map it to the probability of the background category. Key points with probabilities less than the background probability threshold are identified as occluded key points, so as to remove occluded key points from the key points of the target object.

[0176] In other embodiments, occlusion recognition processing for each key point of the target object can also be implemented through the following specific scheme: based on the location of the key point, a region within a preset range is determined in the image to be recognized, and the image of the corresponding region is extracted as the key point image corresponding to the key point; features are extracted from the key point image, and binary classification prediction is performed on the key point image based on the features extracted from the key point image; according to the binary classification prediction result, the key point image is determined to belong to the foreground category or the occlusion category; when the key point image belongs to the occlusion category, the corresponding key point is determined to be an occluded key point, so as to remove the occluded key point from the key points of the target object.

[0177] In some embodiments, occluded key points are removed from the key points of the target object to update the key points of the target object. In step 104, the ratio of the updated key points of the target object to the preset number of key points is used as the degree of completeness of the target object in the image to be recognized.

[0178] For example, when the target object is a human body, after using the second keypoint detection model to perform keypoint recognition processing on the directly cropped human body image, nine keypoints are obtained (nose, left eye, left ear, left shoulder, left elbow, left wrist, left hip, left knee, and left ankle). After occlusion recognition, three of these keypoints (nose, left eye, and left ear) are identified as occluded keypoints. Based on the comparison with the preset 17 keypoints for the target human body, it can be inferred that the target human body is missing at least the right side of its body, and the target human body's face is occluded. To quantify the degree of incompleteness of the target human body, the three occluded keypoints are removed from the identified keypoints. The ratio of the updated number of keypoints (6 keypoints) to the preset number of keypoints (17 keypoints) is used as the degree of completeness of the target object in the image to be recognized. Here, directly comparing the number of keypoints obtained from keypoint recognition with the preset number of keypoints to determine the degree of incompleteness of the target object can quickly determine the completeness of the target object. By performing occlusion recognition on each key point, the actual occlusion situation of the target object is fully considered, and occluded key points are filtered out. When judging the incompleteness of the target object based on the filtered key points, the completeness is considered from two dimensions: the actual absence of key point information of the target object in the image to be recognized and the presence of occlusion of key point information. This makes the judgment of the completeness of the target object more accurate, and it has shown greater application value in application scenarios that need to consider that "the target object is not only in the image, but also needs to be free from occlusion".

[0179] Through the embodiments of this application, explicit target detection is performed on the image to be recognized, extracting images containing target objects. Upon confirming the presence of target objects, subsequent keypoint recognition processing is performed. By comparing the recognized keypoint information with theoretically identified keypoint information, the degree of incompleteness of the target object is accurately and flexibly determined. Compared to simple image classification methods that can only perform binary classification of completeness or incompleteness, this approach enriches the judgment conditions and results, enabling different application values ​​in specific business scenarios. Furthermore, prioritizing the detection of target objects before keypoint recognition processing eliminates misjudgments caused by images not containing target objects, improving the accuracy and precision of incompleteness assessment.

[0180] In some embodiments, the method of steps 101-104 can be applied to the following use cases: when the image to be identified is a candidate cover image of a media account and the completeness of the image to be identified is lower than the completeness threshold, the image to be identified is deleted from the candidate cover image set; when the image to be identified is carried in the information to be recommended and the completeness of the image to be identified is lower than the completeness threshold, the image to be identified is blocked from being recommended or the recommendation weight of the image to be identified is reduced.

[0181] In practical implementation, when the image to be identified is used for selecting the cover image of a media account, if the image to be identified is a candidate cover image of the media account and the completeness of the image to be identified is lower than the completeness threshold, the image to be identified can be deleted from the candidate cover image set through the following specific scheme: Obtain the candidate cover image set of the media account (here, the candidate cover image set can be obtained in any way, such as from media video frames, from an image database related to the target media, or from images or videos uploaded by the media account user, etc.); Obtain the image to be identified from the candidate cover image set, and through steps 101-104 provided in this application embodiment, detect the target object and identify the incompleteness of the target object in the image to be identified to obtain the incompleteness of the target object in the image to be identified; when the completeness is lower than the completeness threshold, determine whether the target object of the image to be identified is incomplete, delete the current image to be identified from the candidate cover image set, and no longer use the corresponding image to be identified as the cover image; when the completeness is higher than the completeness threshold, determine whether the target object of the image to be identified is complete, and the image to be identified can be used as a priority candidate image for the cover image and enter the subsequent selection or cover image processing.

[0182] In practical implementation, when the image to be identified is used in an online recommendation system for image recommendation, if the image to be identified is carried in the information to be recommended and the completeness of the image to be identified is lower than the completeness threshold, the recommended image to be identified or the recommendation weight of the image to be identified can be blocked or reduced through the following specific scheme: Obtain the image to be identified; through steps 101-104 provided in the embodiments of this application, detect the target object of the image to be identified and identify the incompleteness of the target object, and obtain the incompleteness of the target object in the image to be identified; when the completeness is lower than the completeness threshold, determine whether the target object of the image to be identified is incomplete, filter or block the current image to be identified and not recommend it, or reduce the recommendation weight of the image to be identified to reduce the chance of the image to be recommended; when the completeness is higher than the completeness threshold, determine whether the target object of the image to be identified is complete, and the image to be identified can be put into the recommendation pool and enter the subsequent selection or image recommendation.

[0183] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0184] The image recognition method of this application takes human image recognition containing human bodies as an example. It performs target detection on the image to be recognized to obtain an image containing human bodies, performs key point detection on the image containing human bodies to obtain key points and their positions, determines the degree of incompleteness of the human body based on the key points and their positions, and obtains the result of human body incompleteness accurately and flexibly based on pre-set rules. The human body recognition method provided by the embodiments of this application can be applied to the selection of cover images for media accounts or online recommendation systems, etc., as described in detail below.

[0185] See Figure 11 , Figure 11 This is a flowchart illustrating an image recognition method based on artificial intelligence provided in this application embodiment. It specifically shows a method for identifying human body incompleteness in an image, which will be explained below in conjunction with steps 201-205.

[0186] Step 201: Obtain the incomplete human body recognition request.

[0187] In practice, after acquiring an image containing a human body, the system receives an incomplete recognition request for the human body and responds to the request to perform incomplete recognition processing on the image.

[0188] Step 202: Use the human detection model to perform target detection and obtain multiple detection boxes containing different objects in the image to be identified and the positions of the detection boxes.

[0189] Here, the human detection model can be the target detection model provided in the embodiments of this application. For example, the Faster-RCNN model integrates feature extraction, candidate region extraction, detection box regression, and category determination into a single network, resulting in a significant improvement in detection speed. See [link to relevant documentation]. Figure 4 , Figure 4 The process of object detection is shown, which is to take an image to be identified as input and output the detection box of the target region and the category of the target object.

[0190] For example, input the image to be recognized. Figure 4 The convolutional layers (Conv layers) in the image are used to extract image features from the image to be identified. Specifically, a series of convolutions (conv), nonlinear transformations (relu), and pooling operations are used to extract feature maps from the image. Figure 4The Region Proposal Networks (RPNs) in the code are primarily used to generate candidate bounding boxes. Specifically, the RPNs generate multiple anchors, which are then classified and subjected to bounding box regression to produce multiple region proposals. The feature map and the proposed regions are then input into the code. Figure 4 In the Region of Interest (RoI) pooling layer, before entering the RoI pooling layer, candidate boxes are first mapped onto feature maps, resulting in a feature map or feature matrix corresponding to each candidate box. The RoI pooling layer then performs max pooling downsampling on the feature maps corresponding to each candidate box, resulting in candidate feature maps (proposal feature maps) of the same size. Specifically, the input feature maps or feature matrices of different scales are divided into 7 equal parts horizontally and vertically, resulting in 49 regions. Max-downsampling is then performed on each region to obtain multiple candidate feature maps of the same size. These multiple candidate feature maps are then fed into two parallel fully connected layers to determine the target category and perform candidate bounding box position regression, respectively.

[0191] Specifically, Figure 4 The Classifier layer shown is a fully connected layer in one branch, used for multi-class prediction of candidate feature maps. It maps the probability matrix belonging to each class using multiple softmax functions. The other fully connected layer performs candidate bounding box position regression prediction, obtaining the positional offset from the pre-labeled target detection boxes, and adjusting the candidate box positions based on these offsets. Here, combining the classification results, candidate boxes belonging to the same class and with large content overlaps are removed. Combined with candidate bounding box position regression processing, multiple target detection boxes containing different objects and their final precise positions are obtained.

[0192] Step 203: Select the detection box containing the human body, crop the corresponding human body region containing the human body in the image to be identified, obtain the human body image containing a single human body, and perform surrounding filling processing on the target image to obtain the filled human body image.

[0193] In practical implementation, after object detection by the object detection model, the categories and detection boxes of multiple objects in the image to be identified can be obtained. Objects classified as "human" and their corresponding detection boxes are identified. Based on the position of the detection boxes, individual human figures are cropped from the original image at the corresponding positions according to the size of the detection boxes, resulting in multiple human images including individual figures. Each human image is then surrounded by padding. Specifically, the padding method is as follows... Figure 7As shown, for an original human image containing a human body, a ring of uniform pixel values ​​is filled around it to obtain a filled human image.

[0194] Here, the uniform pixel value is a specific pixel value that is set in advance. It can be an empirical pixel value, such as the average pixel value calculated in a large image recognition database (ImageNet).

[0195] Step 204: Use the human body key point model to perform key point detection on the filled human body image to obtain the key points of the human body and their corresponding positions.

[0196] Here, the human body key point model can be the key point recognition model, the first key point detection model, etc. provided in the embodiments of this application.

[0197] In practical implementation, using the human keypoint model as an example, HRNet employs a parallel design approach. During image feature extraction, one image processing branch maintains high resolution. Image processing is divided into several stages. At the beginning of each stage, a smaller resolution feature map (also smaller in scale) is added, creating a branch of the corresponding resolution. Through interpolation upsampling and convolution downsampling, it is ensured that the feature map of each branch at the beginning of each stage combines the features of different scale feature maps from the previous stage. After multiple stages of processing, all small-scale feature maps are interpolated and upsampled to obtain a feature map of the same size as the largest-scale feature map (which always maintains high resolution). The feature map is mapped to probability maps of different channels. Each channel's probability map corresponds to the probability distribution of a keypoint in a preset human keypoint set. The location of the maximum value in each channel is taken as the location of the keypoint predicted for that channel. Combining the prediction results of all channels yields all keypoints and their location information corresponding to the preset keypoint set.

[0198] Here, each preset set of key points for the human body is defined to include 17 key points, see [link to relevant documentation]. Figure 10 , Figure 10 The diagram shows 17 key points of the human body: the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

[0199] It should be noted that most human keypoint detection models first downsample to reduce resolution and then upsample to increase resolution, thus losing some spatial information and causing quantization errors in the final recognition calculation. To obtain a higher resolution feature map, HRNet uses a parallel design method that maintains a single high-resolution branch throughout, preserving rich spatial information and reducing quantization errors in the detection results.

[0200] Step 205: Determine the degree of incompleteness of the human body based on preset rules.

[0201] In practice, if the detected keypoints are located in the surrounding area after filling, it is determined that the keypoints of the current human body part have been cropped and are not within the original human body image. The location and region of each keypoint are then determined sequentially to see if any detected keypoint is within the surrounding area after filling or within the original human body image. Based on the information from all keypoints, the degree of incompleteness of the human body is determined according to preset rules.

[0202] Here, preset rules can be used to quantify the degree of incompleteness of the human body. For example, the ratio of the number of key points of the target object in the original human body image to the preset number of key points can be used as the degree of completeness of the target object in the image to be identified.

[0203] In practice, the more key points detected are in the surrounding area after filling, the greater the degree of incompleteness of the human body. Conversely, the more key points detected are in the original image area, the less incomplete the human body. Based on the degree of incompleteness, an incompleteness threshold is set. When the degree of incompleteness is higher than the degree of incompleteness threshold, the human body is determined to be complete; otherwise, the human body is incomplete.

[0204] Because conventional image classification methods can only output a binary classification result of whether an image is complete, without making an explicit judgment on whether there is a human body in the image, it may lead to misclassification of images that do not contain human bodies. Furthermore, image classification models cannot obtain an intermediate result of the degree of human body incompleteness, making it difficult to meet the needs of different business scenarios. Once the definition of human body incompleteness changes in the business scenario, the model needs to be retrained, which lacks flexibility.

[0205] The incomplete human body recognition method based on key point detection provided in this application first determines whether there is a human body in the image to be recognized, avoiding misjudgment of the integrity of images that do not contain human bodies. Furthermore, based on whether the key points are in the surrounding area after filling, the degree of human body incompleteness can be obtained. A threshold is set based on the degree of human body incompleteness to make the final judgment on human body integrity. By adjusting the preset rules and integrity threshold in different business scenarios, it can be flexibly applied to image incompleteness recognition in various business scenarios.

[0206] See Figure 12 , Figure 12 This is a schematic diagram illustrating a process of applying the image recognition method provided in this application to the selection of a media account cover image. The explanation will focus on steps 301-305 below.

[0207] Step 301: Obtain text / image / video content.

[0208] Specifically, the selection of cover images for media accounts can be applied to Tencent Video Channel. For example, when displaying an article / video about sports on an online video client, if the current sports information needs to display information about athletes or event hosts, the cover image of the information must at least ensure that the human body information of the athletes or event hosts is complete.

[0209] In practice, we obtain the text / image / video content uploaded by media account users. Here, the text / image / video content provides cover image materials.

[0210] Step 302: Obtain multiple candidate cover images.

[0211] In practice, multiple cover image candidates can be obtained from text / image / video content in any way. For example, they can be selected or cropped from text / image materials, or cropped from video frames.

[0212] Step 303: Select a candidate cover image, perform incomplete human body recognition processing on the candidate cover image, and determine whether the human body contained in the candidate image is complete.

[0213] Here, the target detection model and key point recognition model provided in the embodiments of this application are used to perform human body recognition and incompleteness detection on the candidate cover image to obtain the recognition result of whether the human body is complete. If the human body is complete, proceed to step 305; if the human body is incomplete, proceed to step 304.

[0214] Step 304: Remove the current cover image candidate from multiple cover image candidate images.

[0215] Step 305: Process the candidate cover images.

[0216] Here, the candidate cover image is processed. The current candidate cover image can be used as the priority candidate image for the cover image and then enter the subsequent selection process or directly enter the cover image processing stage.

[0217] See Figure 13 , Figure 13 This is a schematic flowchart illustrating the application of the image recognition method provided in this application to a recommendation system. The explanation will focus on steps 401-404 below.

[0218] Step 401: Obtain text / image / video content.

[0219] Specifically, recommendation systems can be applied to QQ Kandian and WeChat official accounts to recommend images that users may be interested in. For example, when recommending sports event images to sports enthusiasts, if the current sports event image needs to display information about athletes or event hosts, at least the complete human body information of the athletes or event hosts in the image should be ensured.

[0220] Here, the text / video content can be content pre-stored in the recommendation pool of the recommendation system, or it can be content uploaded by system users in real time, etc.

[0221] Step 402: Select an image to be recommended, perform incomplete human body recognition processing on the image to determine whether the human body contained in the image is complete.

[0222] Here, the target detection model and key point recognition model provided in the embodiments of this application are used to perform human body recognition and incompleteness detection on the recommended image to obtain the recognition result of whether the human body is complete. If the human body is complete, proceed to step 404; if the human body is incomplete, proceed to step 403.

[0223] Step 403: Disable or block the currently recommended image, or reduce the weight of the recommended image in the recommendation process.

[0224] Step 404: Place the image to be recognized into the recommendation pool and wait for recommendation processing.

[0225] The human body incompleteness recognition method provided in this application embodiment determines whether a human body exists in the image to be identified, avoiding misjudgments of the integrity of images that do not contain human bodies. Furthermore, by determining whether key points are located in the surrounding area after filling, the degree of human body incompleteness can be quickly and flexibly determined. In practical applications, the human body incompleteness recognition method provided in this application embodiment can provide higher-quality candidate images for cover image selection and higher-quality content for online recommendation systems, significantly improving the user experience.

[0226] The following description continues to illustrate the exemplary structure of the artificial intelligence-based image recognition device 255 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2As shown, the software modules stored in the AI-based image recognition device 255 in the memory 250 may include: a target detection module 2551, used to perform target detection processing on the image to be recognized in order to obtain the target object image from the image to be recognized; a feature extraction module 2552, used to perform feature extraction processing on the target object image to obtain the corresponding image features; a key point recognition module 2553, used to perform key point recognition processing on the image features to obtain the key points of the target object and their corresponding positions; and an integrity discrimination module 2554, used to determine the integrity of the target object in the image to be recognized based on the key points of the target object and their corresponding positions.

[0227] In some embodiments, the target detection module 2551 is further configured to: perform target detection processing on the image to be identified to obtain a detection box including the target object; and crop the target object image from the image to be identified based on the position of the detection box.

[0228] In some embodiments, the target detection module 2551 is further configured to: extract features from the image to be identified to obtain a corresponding feature map; determine multiple candidate boxes in the image to be identified; map the candidate boxes onto the feature map to obtain multiple candidate feature maps; perform max pooling on the multiple candidate feature maps to obtain multiple candidate region maps of the same size; and perform classification processing and candidate box position regression processing on the multiple candidate region maps to obtain a detection box including the target object.

[0229] In some embodiments, the feature extraction module 2552 is further configured to: fill the surrounding area of ​​the target object image, perform feature extraction on the filled target object image to obtain corresponding image features, or directly perform feature extraction on the cropped target object image to obtain corresponding image features.

[0230] In some embodiments, when the image features are extracted from a padded target object image, the key point recognition module 2553 is further configured to: invoke a first key point detection model to perform the following processing: map the image features into a probability map of multiple channels, wherein each channel's probability map corresponds to the probability distribution of a key point in a preset key point set, and the probability distribution is used to characterize the probability that each pixel in the target object image belongs to the key point corresponding to the probability map; perform the following processing for each probability map: identify the pixel with the highest probability in the probability map as the key point corresponding to the probability map, and identify the position of the pixel with the highest probability as the position of the key point corresponding to the probability map; combine the key points and corresponding positions identified from each probability map to form a key point recognition result for the target object; wherein the key point recognition result includes multiple key points and corresponding positions, and the multiple key points correspond one-to-one with all key points in the preset key point set.

[0231] In some embodiments, when the image features are extracted from a padded target object image, the key point recognition module 2553 is further configured to: invoke a second key point detection model to perform the following processing: map the image features into a probability map, wherein the probability map includes the probability of each pixel in the target object image corresponding to each key point in a preset key point set; perform the following processing for each pixel in the target object image: determine the maximum probability among the probabilities of each key point in the preset key point set; when the maximum probability exceeds a probability threshold, identify the pixel as the key point corresponding to the maximum probability, and identify the position of the pixel as the position of the key point corresponding to the maximum probability; combine the key points identified from the probability map and their corresponding positions to form a key point recognition result for the target object; wherein the key point recognition result includes at least one key point in the preset key point set and its corresponding position.

[0232] In some embodiments, when the image features are extracted from a target object image that has undergone padding, the completeness determination module 2554 is further configured to: perform the following processing for each key point of the target object: when the key point is located in the surrounding area used for padding, determine that the key point is a missing key point in the image to be identified; remove the missing key points from the identified key points of the target object to update the identified key points of the target object; and use the ratio of the updated number of identified key points of the target object to a preset number of key points as the completeness of the target object in the image to be identified; wherein, the preset number of key points is the key point count of a preset set of key points of the target object.

[0233] In some embodiments, when the image features are extracted from the image of the direct target object, the integrity discrimination module 2554 is further configured to: use the ratio of the number of key points of the target object obtained by key point recognition processing to the preset number of key points as the integrity of the target object in the image to be recognized; wherein, the preset number of key points is the key point count of the preset key point set of the target object.

[0234] In some embodiments, before determining the ratio, the integrity discrimination module 2554 is further configured to: perform occlusion recognition processing on each key point of the target object to determine the occluded key points in the key points of the target object image; and remove the occluded key points in the key points of the target object to update the key points of the target object.

[0235] In some embodiments, the AI-based image recognition device further includes: a processing module 2555, configured to delete the image to be recognized from the candidate cover image set when the image to be recognized is a candidate cover image of a media account and the completeness of the image to be recognized is lower than the completeness threshold; and to block the recommendation of the image to be recognized or reduce the recommendation weight of the image to be recognized when the image to be recognized is carried in the information to be recommended and the completeness of the image to be recognized is lower than the completeness threshold.

[0236] It should be noted that the description of the image recognition device based on artificial intelligence in this application is similar to the description of the image recognition method based on artificial intelligence described above, and has similar beneficial effects as the image recognition method based on artificial intelligence, so it will not be repeated.

[0237] This application provides an electronic device, including:

[0238] Memory, used to store executable instructions;

[0239] The processor, when executing executable instructions stored in the memory, implements the artificial intelligence-based image recognition method provided in the embodiments of this application.

[0240] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the artificial intelligence-based image recognition method provided in this application.

[0241] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the artificial intelligence-based image recognition method described above in this application.

[0242] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application. For example... Figure 3A , 3B The method shown.

[0243] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0244] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0245] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0246] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0247] In summary, through the embodiments of this application, explicit target detection is performed on the image to be identified, and the image containing the target object is extracted for key point recognition processing of the target object. By comparing the identified key point information with the theoretically identified key point information, the degree of incompleteness of the target object is accurately and flexibly judged, improving the accuracy and precision of incompleteness judgment, and enabling different application values ​​in specific business scenarios.

[0248] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image recognition method based on artificial intelligence, characterized in that, include: Target detection processing is performed on the image to be identified in order to obtain the image of the target object from the image to be identified; The surrounding area of ​​the target object image is filled, and the target object image of a preset size after the filling process is subjected to feature extraction processing to obtain the corresponding image features; The first keypoint detection model is invoked to perform the following processing: The image features are mapped into a probability map of multiple channels, wherein the probability map of each channel corresponds to the probability distribution of a key point in a preset key point set. The probability distribution is used to characterize the probability that each pixel in the target object image belongs to the key point corresponding to the probability map. The first key point detection model is trained through a first training sample set, which includes occluded image samples and unoccluded image samples. The occluded image samples and the unoccluded image samples carry key point location markers. Some key points of the target object in the occluded image samples are occluded by the background. For each of the probability maps, the following processing is performed: the pixel with the highest probability in the probability map is identified as the key point corresponding to the probability map, and the position of the pixel with the highest probability is identified as the position of the key point corresponding to the probability map; The key points and their corresponding positions identified from each probability map are combined to form the key point identification result of the target object. The key point identification result includes multiple key points and their corresponding positions, and the multiple key points correspond one-to-one with all key points in the preset key point set. Based on the key points and corresponding positions of the target object, the degree of completeness of the target object in the image to be identified is determined.

2. The method according to claim 1, characterized in that, The step of performing target detection processing on the image to be identified, in order to obtain a target object image from the image to be identified, includes: The image to be recognized is processed for target detection to obtain a detection bounding box that includes the target object. Based on the position of the detection box, the target object image is cropped from the image to be identified; The method further includes: The cropped target object image is directly subjected to feature extraction processing to obtain the corresponding image features.

3. The method according to claim 2, characterized in that, The process of performing target detection processing on the image to be recognized to obtain a detection bounding box including the target object includes: Extract the features of the image to be identified to obtain the corresponding feature map; Multiple candidate bounding boxes are determined in the image to be identified; The candidate bounding boxes are mapped onto the feature maps to obtain multiple corresponding candidate feature maps; Max pooling is performed on the multiple candidate feature maps to obtain multiple candidate region maps of the same size; The multiple candidate region maps are classified and candidate box position regression is performed to obtain a detection box including the target object.

4. The method according to claim 1, characterized in that, The method further includes: The second keypoint detection model is invoked to perform the following processing: The image features are mapped to a probability map, wherein the probability map includes the probability of each pixel in the target object image corresponding to each key point in a preset set of key points; Perform the following processing on each pixel in the target object image: Determine the maximum probability among the probabilities of the pixel corresponding to each key point in the preset key point set; When the maximum probability exceeds the probability threshold, the pixel is identified as the key point corresponding to the maximum probability, and the position of the pixel is identified as the position of the key point corresponding to the maximum probability. The key points and their corresponding positions identified from the probability map are combined to form the key point identification result of the target object; The key point identification result includes at least one key point and its corresponding position in the preset key point set.

5. The method according to claim 1, characterized in that, Determining the completeness of the target object in the image to be identified based on the key points and corresponding positions of the target object includes: Perform the following processing for each key point of the target object: When the location of the key point is within the surrounding area used for the filling process, the key point is determined to be a missing key point in the image to be identified. Remove the missing key points from the identified key points of the target object to update the identified key points of the target object; The ratio of the number of key points of the target object obtained after the update to the preset number of key points is used as the degree of completeness of the target object in the image to be identified. The preset number of key points is the key point count of the preset key point set of the target object.

6. The method according to claim 1, characterized in that, The method further includes: The ratio of the number of key points of the target object obtained by the key point recognition process to the preset number of key points is used as the degree of completeness of the target object in the image to be recognized. The preset number of key points is the key point count of the preset key point set of the target object.

7. The method according to claim 5 or 6, characterized in that, Before determining the ratio, the method further includes: Occlusion recognition processing is performed on each key point of the target object to determine the occluded key points in the key points of the target object image; Remove the occluded key points from the key points of the target object to update the key points of the target object.

8. The method according to any one of claims 1 to 6, characterized in that, The method further includes: When the image to be identified is a candidate cover image for a media account, and the completeness of the image to be identified is lower than the completeness threshold, the image to be identified is removed from the candidate cover image set; When the image to be identified is carried in the information to be recommended, and the completeness of the image to be identified is lower than the completeness threshold, the image to be identified is blocked from being recommended, or the recommendation weight of the image to be identified is reduced.

9. An image recognition device based on artificial intelligence, characterized in that, include: The target detection module is used to perform target detection processing on the image to be identified, so as to obtain the target object image from the image to be identified; The feature extraction module is used to fill the surrounding area of ​​the target object image, and to perform feature extraction on the target object image of a preset size after the filling process to obtain the corresponding image features. The key point recognition module is used to call the first key point detection model to perform the following processing: mapping the image features into a probability map of multiple channels, wherein the probability map of each channel corresponds to the probability distribution of a key point in a preset key point set, and the probability distribution is used to characterize the probability that each pixel in the target object image belongs to the key point corresponding to the probability map. The first key point detection model is trained through a first training sample set, which includes occluded image samples and unoccluded image samples. The occluded image samples and the unoccluded image samples carry key point location markers. Some key points of the target object in the occluded image samples are occluded by the background. For each probability map, the following processing is performed: the pixel with the highest probability in the probability map is identified as the key point corresponding to the probability map, and the position of the pixel with the highest probability is identified as the position of the key point corresponding to the probability map; the key points and corresponding positions identified from each probability map are combined to form the key point identification result of the target object, wherein the key point identification result includes multiple key points and corresponding positions, and the multiple key points correspond one-to-one with all key points in the preset key point set; The integrity determination module is used to determine the degree of integrity of the target object in the image to be identified based on the key points of the target object and their corresponding positions.

10. The apparatus as claimed in claim 9, characterized in that, The target detection module is also used for: The image to be identified is subjected to target detection processing to obtain a detection box including the target object; based on the position of the detection box, the target object image is cropped from the image to be identified; The feature extraction module is also used to directly perform feature extraction processing on the cropped target object image to obtain the corresponding image features.

11. The apparatus as claimed in claim 10, characterized in that, The target detection module is also used for: Features of the image to be identified are extracted to obtain a corresponding feature map; multiple candidate boxes are determined in the image to be identified; the candidate boxes are mapped to the feature map to obtain multiple candidate feature maps; max pooling is performed on the multiple candidate feature maps to obtain multiple candidate region maps of the same size; classification and candidate box position regression are performed on the multiple candidate region maps to obtain a detection box including the target object.

12. The apparatus as claimed in claim 9, characterized in that, The key point recognition module is also used for: The second keypoint detection model is invoked to perform the following processing: the image features are mapped to a probability map, wherein the probability map includes the probability of each pixel in the target object image corresponding to each keypoint in a preset keypoint set; for each pixel in the target object image, the following processing is performed: the maximum probability among the probabilities of the pixel corresponding to each keypoint in the preset keypoint set is determined; when the maximum probability exceeds a probability threshold, the pixel is identified as the keypoint corresponding to the maximum probability, and the position of the pixel is identified as the position of the keypoint corresponding to the maximum probability; the keypoints identified from the probability map and their corresponding positions are combined to form the keypoint recognition result of the target object; wherein the keypoint recognition result includes at least one keypoint and its corresponding position in the preset keypoint set.

13. The apparatus as claimed in claim 9, characterized in that, The integrity determination module is also used for: For each key point of the target object, the following processing is performed: when the key point is located in the surrounding area used for the filling process, the key point is determined to be a missing key point in the image to be identified; Remove the missing key points from the identified key points of the target object to update the identified key points of the target object; The ratio of the number of key points of the target object obtained after the update to the preset number of key points is used as the completeness of the target object in the image to be identified; wherein, the preset number of key points is the key point count of the preset key point set of the target object.

14. The apparatus as claimed in claim 9, characterized in that, The integrity determination module is also used for: The ratio of the number of key points of the target object obtained by the key point recognition process to the preset number of key points is used as the degree of completeness of the target object in the image to be recognized; wherein, the preset number of key points is the key point count of the preset key point set of the target object.

15. The apparatus as claimed in claim 13 or 14, characterized in that, Before determining the ratio, the integrity discrimination module is further configured to: Occlusion recognition processing is performed on each key point of the target object to determine the occluded key points in the key points of the target object image; Remove the occluded key points from the key points of the target object to update the key points of the target object.

16. The apparatus according to any one of claims 9 to 14, characterized in that, It also includes a processing module for: When the image to be identified is a candidate cover image for a media account, and the completeness of the image to be identified is lower than the completeness threshold, the image to be identified is removed from the candidate cover image set; when the image to be identified is carried in the information to be recommended, and the completeness of the image to be identified is lower than the completeness threshold, the image to be identified is blocked from being recommended, or the recommendation weight of the image to be identified is reduced.

17. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the method of any one of claims 1 to 8 when executed by a processor.

19. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images