Method, device, equipment and storage medium for detecting associated objects in images

Abstract

The present application proposes a method, device, equipment and storage medium for detecting associated objects in an image. The method includes detecting a face object, a preset human body part object and a human hand object contained in an image. For each two objects among the above-mentioned face objects, preset human body part objects and human hand objects, a correlation prediction is performed. The human body objects contained in the above-mentioned image are segmented, and based on the areas corresponding to each of the above-mentioned human body objects obtained by segmentation, the human body objects to which the above-mentioned face objects and the above-mentioned preset human body part objects respectively belong are determined. Based on the human body objects to which the above-mentioned face objects and the above-mentioned preset human body part objects respectively belong, the first correlation prediction results between the above-mentioned face objects and the above-mentioned preset human body part objects are adjusted, and the associated objects in the above-mentioned image are determined according to the adjusted first correlation prediction results.

Description

[0001] Cross-references to related publications

[0002] This disclosure claims priority to Singapore patent publication No. 10202013267T filed on December 31, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to computer technology, and in particular to a method, apparatus, device and storage medium for detecting associated objects in an image. Background Art

[0004] Intelligent video analysis technology can help humans understand the state of objects in physical space and the relationship between objects. In one application scenario of intelligent video analysis, it is necessary to identify the identity of the person corresponding to the human body part appearing in the video.

[0005] Specifically, the easily identifiable first human body part can be matched one by one with the identity of the person. After the above correspondence is maintained, the first human body part that is an associated object with the second human body part appearing in the video can be determined, and the identity of the person corresponding to the second human body part can be identified based on the determined first human body part. Among them, the associated object can refer to the first human body part and the second human body part belonging to the same person. If two human body parts are associated objects with each other, it can be considered that the above human body parts belong to the same person.

[0006] By associating the human body parts in the image, it can further help analyze the behavior and status of individuals in a multi-person scene, as well as the relationship between multiple people. For example, from multiple face objects and multiple hand objects detected in the image, the face objects and hand objects belonging to the same person can be determined. After the face objects and hand objects belonging to the same person are determined, the identity information of the person who performs a preset action or touches a preset area through the hand object can be determined based on the face objects associated with the hand objects. Summary of the invention

[0007] In view of this, the present application discloses a method for detecting associated objects in an image, the method comprising: detecting a face object, a preset human body part object and a hand object contained in the image; wherein the preset human body part object represents a preset human body part in the body connection part between the face and the hand; performing association prediction on each two of the face objects, the preset human body part objects and the hand objects to obtain a first association prediction result between the face object and the preset human body part object, a second association prediction result between the preset human body part object and the hand object, and a third association prediction result between the face object and the hand object; segmenting the human body objects contained in the image, and determining the first and second human body objects to which the face object and the preset human body part object belong based on the areas corresponding to each of the human body objects obtained by the segmentation; adjusting the first association prediction result based on the first and second human body objects, and determining the associated objects in the image based on the adjusted first, second and third association prediction results.

[0008] In some of the illustrated embodiments, the facial objects, preset human body part objects and human hand objects contained in the above-mentioned detection image include: a first bounding box of the facial objects and a second bounding box of the preset human body part objects in the detection image; the above-mentioned areas corresponding to each of the above-mentioned human body objects obtained based on the segmentation, determining the first human body object and the second human body object to which the above-mentioned facial objects and the above-mentioned preset human body part objects belong respectively, including: determining a first overlapping area between the area corresponding to the facial object and the area corresponding to each of the above-mentioned human body objects according to the above-mentioned first bounding box; determining a second overlapping range between the area corresponding to the above-mentioned preset human body part object and the area corresponding to each of the above-mentioned human body objects according to the second bounding box; and determining the first human body object to which the above-mentioned facial object belongs and the first human body object and the second human body object to which the above-mentioned preset human body part objects belong respectively according to the above-mentioned first overlapping range and the second overlapping range.

[0009] In some of the illustrated embodiments, the above-mentioned determining the first human body object and the second human body object to which the above-mentioned facial object and the above-mentioned preset human body part object respectively belong based on the first overlapping range and the second overlapping range includes: determining a first target area with the largest first overlapping range with the area corresponding to the above-mentioned facial object among the areas corresponding to each of the above-mentioned human body objects; determining a second target area with the largest second overlapping range with the area corresponding to the above-mentioned preset human body part object among the areas corresponding to each of the above-mentioned human body objects; determining the human body object corresponding to the above-mentioned first target area as the first human body object to which the above-mentioned facial object belongs; and determining the human body object corresponding to the above-mentioned second target area as the second human body object to which the above-mentioned preset human body part object belongs.

[0010] In some of the illustrated embodiments, the first correlation prediction result is adjusted based on the first human body object and the second human body object to which the above-mentioned face object and the above-mentioned preset human body part object respectively belong, including: matching the first human body object with the second human body object to obtain a matching result; and adjusting the first correlation prediction result based on the above-mentioned matching result.

[0011] In some of the illustrated embodiments, the first correlation prediction result is adjusted based on the matching result, including: when the first human body object matches the second human body object, increasing the correlation prediction score in the first correlation prediction result; and / or, when the first human body object does not match the second human body object, reducing the correlation prediction score in the first correlation prediction result.

[0012] In some of the illustrated embodiments, the method further includes: combining the detected facial objects, the preset human body part objects and the human hand objects to generate at least one triple, wherein each triple includes a facial object, a human body part object and a hand object; performing correlation prediction on every two objects among the facial objects, the preset human body part objects and the human hand objects, including: performing correlation prediction on every two objects in each triple; determining the associated objects in the image based on the adjusted first correlation prediction result, the second correlation prediction result and the third correlation prediction result, including: determining the fourth correlation prediction result corresponding to each triple based on the adjusted first correlation prediction result, the second correlation prediction result and the third correlation prediction result; determining the associated objects in the image based on the fourth correlation prediction result corresponding to each triple.

[0013] In some of the illustrated embodiments, the above-mentioned determination of associated objects in the above-mentioned image based on the fourth association prediction result corresponding to each triplet includes: determining the current triplet for each triplet in turn according to the order of the association prediction scores in the above-mentioned fourth association prediction results corresponding to each triplet from high to low, and executing: based on the determined associated objects, determining whether the number of associated hand objects that are mutually associated objects with the face objects included in the current triplet reaches a first preset threshold, and determining whether the number of associated face objects that are mutually associated objects with the hand objects included in the current triplet reaches a second preset threshold; if the number of associated hand objects does not reach the above-mentioned first preset threshold, and the number of associated face objects does not reach the above-mentioned second preset threshold, then the face object and hand object included in the current triplet are determined as associated objects in the above-mentioned image.

[0014] In some of the illustrated embodiments, the preset human body part object includes at least one of a shoulder object and an elbow object.

[0015] In some of the illustrated embodiments, the method further includes: outputting a detection result of an associated object in the image.

[0016] In some of the illustrated embodiments, the above-mentioned detection of the face object, the preset human body part object and the human hand object contained in the image includes: using an object detection model constructed based on a neural network to detect the face object, the preset human body part object and the human hand object contained in the image. The above-mentioned prediction of the association between each two objects among the above-mentioned face objects, the preset human body part objects and the human hand objects includes: using a correlation prediction model constructed based on a neural network to predict the association between each two objects among the above-mentioned face objects, the preset human body part objects and the human hand objects; the above-mentioned segmentation of the human body objects contained in the above-mentioned image includes: using an instance segmentation model constructed based on a neural network to segment the human body objects contained in the above-mentioned image.

[0017] In some of the illustrated embodiments, the method further includes: training the object detection model based on a first training sample set; wherein the first training sample set includes image training samples with first annotation information; the first annotation information includes bounding boxes of face objects, preset body part objects and hand objects; training the instance segmentation model based on a second training sample set; wherein the second training sample set includes image training samples with second annotation information; the second annotation information includes bounding boxes of human objects; jointly training the object detection model, the instance segmentation model and the correlation prediction model based on a third training sample set; wherein the third training sample set includes image training samples with third annotation information; the third annotation information includes bounding boxes corresponding to face objects, preset body part objects, hand objects and human objects, respectively, as well as correlation annotation information between face objects and preset body part objects, correlation annotation information between preset body part objects and hand objects, and correlation annotation information between face objects and hand objects.

[0018] The present application also proposes a detection device for associated objects in an image, the device comprising: an object detection module, used to detect a face object, a preset human body part object and a hand object contained in the image; wherein the preset human body part object represents a preset human body part in the body connection part between the face and the hand; a correlation prediction module, used to perform correlation prediction on each two of the face objects, the preset human body part objects and the hand objects, and obtain a first correlation prediction result between the face object and the preset human body part object, a second correlation prediction result between the preset human body part object and the hand object, and a third correlation prediction result between the face object and the hand object; an attribution determination module, used to segment the human body objects contained in the image, and determine the first human body object and the second human body object to which the face object and the preset human body part object belong respectively based on the regions corresponding to each of the human body objects obtained by segmentation; and a correlated object determination module, used to adjust the first correlation prediction result based on the first human body object and the second human body object, and determine the associated objects in the image according to the adjusted first correlation prediction result, the second correlation prediction result and the third correlation prediction result.

[0019] In some of the illustrated embodiments, the object detection module is specifically used to: detect a first bounding box and a second bounding box corresponding to a facial object and a preset human body part object in an image, respectively; the attribution relationship determination module includes: an overlap range determination module, used to determine a first overlap range between an area corresponding to the facial object and an area corresponding to each of the human body objects according to the first bounding box, and to determine a second overlap range between an area corresponding to the preset human body object and an area corresponding to each of the human body objects according to the second bounding box; an attribution relationship determination submodule, used to determine the first human body object and the second human body object to which the facial object and the preset human body part object belong, respectively, according to the first overlap range and the second overlap range.

[0020] In some of the illustrated embodiments, the above-mentioned attribution relationship determination submodule is specifically used to: determine a first target area with the largest first overlap range with the area corresponding to the above-mentioned facial object among the areas corresponding to the above-mentioned human objects respectively; determine a second target area with the largest second overlap range with the area corresponding to the above-mentioned preset human body part object among the areas corresponding to the above-mentioned human objects respectively; determine that the human body object corresponding to the above-mentioned first target area is the first human body object to which the facial object belongs; and determine that the human body object corresponding to the above-mentioned second target area is the second human body object to which the above-mentioned preset human body part object belongs.

[0021] In some of the illustrated embodiments, the associated object determination module is specifically used to: match the first human object with the second human object to obtain a matching result; and adjust the first association prediction result based on the matching result.

[0022] In some of the illustrated embodiments, the associated object determination module is used to: increase the association prediction score in the first association prediction result when the first human body object matches the second human body object; and / or reduce the association prediction score in the first association prediction result when the first human body object does not match the second human body object.

[0023] In some of the illustrated embodiments, the apparatus further includes: a combination module for combining the detected face object, the preset human body part object and the hand object to generate at least one triple, wherein each triple includes a face object, a human body part object and a hand object. The correlation prediction module is specifically used to: perform correlation prediction on every two objects in each triple; the associated object determination module includes: a triple correlation relationship prediction module for determining the fourth correlation prediction result corresponding to each triple based on the adjusted first correlation prediction result, the second correlation prediction result and the third correlation prediction result; an associated object determination submodule for determining the associated objects in the image based on the fourth correlation prediction result corresponding to each triple.

[0024] In some of the illustrated embodiments, the associated object determination submodule is specifically used to: determine the current triplet for each triplet in turn according to the order of the association prediction scores in the fourth association prediction results corresponding to each triplet from high to low, and execute: based on the determined associated objects, determine whether the number of associated hand objects that are mutually associated with the face objects included in the current triplet reaches a first preset threshold, and determine whether the number of associated face objects that are mutually associated with the hand objects included in the current triplet reaches a second preset threshold; if the number of associated hand objects does not reach the first preset threshold, and the number of associated face objects associated with the hand objects included in the current triplet does not reach the second preset threshold, then the face object and hand object included in the current triplet are determined as associated objects in the above image.

[0025] In some of the illustrated embodiments, the preset human body part object includes at least one of a shoulder object and an elbow object.

[0026] In some of the illustrated embodiments, the apparatus further includes: an output module, configured to output a detection result of the associated object in the image.

[0027] In some of the illustrated embodiments, the object detection module is specifically used to: use an object detection model constructed based on a neural network to detect facial objects, preset human body part objects and human hand objects contained in the image; the association prediction module is specifically used to: use an association prediction model constructed based on a neural network to perform association prediction on every two of the facial objects, preset human body part objects and human hand objects; the attribution relationship determination module is specifically used to: use an instance segmentation model constructed based on a neural network to segment the human body objects contained in the image.

[0028] In some of the illustrated embodiments, the apparatus further includes: a first training module for training the object detection model based on a first training sample set; wherein the first training sample set includes image training samples with first annotation information; the first annotation information includes bounding boxes of face objects, preset body part objects, and hand objects; a second training module for training the instance segmentation model based on a second training sample set; wherein the second training sample set includes image training samples with second annotation information; the second annotation information includes bounding boxes of human objects; a joint training module for jointly training the object detection model, the instance segmentation model, and the correlation prediction model based on a third training sample set; wherein the third training sample set includes image training samples with third annotation information; the third annotation information includes bounding boxes corresponding to face objects, preset body part objects, hand objects, and human objects, respectively, as well as correlation annotation information between face objects and preset body part objects, correlation annotation information between preset body part objects and hand objects, and correlation annotation information between face objects and hand objects.

[0029] The present application also proposes an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the executable instructions stored in the memory to implement a method for detecting associated objects in an image as shown in any of the above embodiments.

[0030] The present application also proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the method for detecting associated objects in an image as shown in any of the above embodiments.

[0031] The present application also proposes a computer program product, including a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the method for detecting associated objects in an image as shown in any of the above embodiments.

[0032] In the scheme of the above method, on the one hand, by using a preset human body part closely related to the human hand object as an intermediary, the first correlation prediction result between the human face object and the preset human body part, the second correlation prediction result between the preset human body part and the human hand object, and the third correlation prediction result between the human face object and the human hand object are determined respectively. Then, based on the above first correlation prediction result, the above second correlation prediction result, and the above third correlation prediction result, the associated object in the above image is determined, so that less interference information is introduced in the process of determining the associated object, thereby ensuring the accuracy of the determination result of the associated human face object and human hand object.

[0033] On the other hand, by optimizing the first correlation prediction result between the face object and the preset body part based on the result of segmenting the human body object contained in the above image, the correlation prediction result between the face object and the preset body part is made more accurate, thereby improving the accuracy of the correlation object determination result.

[0034] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 This is a flow chart of a method for detecting associated objects in an image shown in the present application;

[0037] Figure 2 A schematic diagram of a method for detecting associated objects shown in the present application;

[0038] Figure 3 A schematic diagram of the object detection process shown in this application;

[0039] Figure 4 This is a schematic diagram of the correlation prediction process shown in this application;

[0040] Figure 5 A schematic diagram of a correlation prediction process shown in this application;

[0041] Figure 6 This is a schematic diagram of the image segmentation process shown in this application;

[0042] Figure 7A schematic diagram of a flow chart of a model training method according to an embodiment of the present application;

[0043] Figure 8 A structural diagram of a detection device for associated objects in an image shown in the present application;

[0044] Fig. 9 This is a hardware structure diagram of an electronic device shown in this application. DETAILED DESCRIPTION

[0045] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0046] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "above", and "the" used in this application and the appended claims are also intended to include plural forms, unless other meanings are clearly indicated below. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determination".

[0047] The present application aims to propose a method for detecting associated objects in an image (hereinafter referred to as the associated object detection method). On the one hand, the method uses a preset human body part that is closely related to a human hand object as an intermediary to respectively determine a first correlation prediction result between a face object and the preset human body part, a second correlation prediction result between the preset human body part and a human hand object, and a third correlation prediction result between a face object and a human hand object. Then, based on the above-mentioned first correlation prediction result, the above-mentioned second correlation prediction result, and the above-mentioned third correlation prediction result, the associated objects in the above-mentioned image are determined, so that in the process of determining the associated objects, less interference information is introduced, thereby ensuring the accuracy of the determination results of the associated face objects and human hand objects.

[0048] On the other hand, the method optimizes the first correlation prediction result between the face object and the preset human body part by segmenting the human body object contained in the above image, thereby making the correlation prediction result between the face object and the preset human body part more accurate, thereby improving the accuracy of the associated object determination result.

[0049] See also Figure 1 , Figure 1 This is a flow chart of a method for detecting associated objects in an image shown in the present application.

[0050] like Figure 1 As shown, the above method may include:

[0051] S102, detecting a face object, a preset body part object and a hand object contained in an image; wherein the preset body part object represents a preset body part in a body connection portion between a face and a hand.

[0052] S104, performing correlation prediction on every two objects among the above-mentioned face objects, preset human body part objects and human hand objects, and obtaining a first correlation prediction result between the above-mentioned face objects and the above-mentioned preset human body part objects, a second correlation prediction result between the above-mentioned preset human body part objects and the above-mentioned hand objects, and a third correlation prediction result between the above-mentioned face objects and the above-mentioned hand objects.

[0053] S106, segmenting the image to determine at least one human object, and determining the first human object and the second human object to which the face object and the preset human body part object belong respectively based on the area corresponding to each of the at least one human object obtained by segmentation.

[0054] S108, adjusting the first correlation prediction result based on the first human object and the second human object, and determining the associated object in the image according to the adjusted first correlation prediction result, the second correlation prediction result and the third correlation prediction result.

[0055] The above-mentioned associated object detection method can be applied to electronic devices. Among them, the above-mentioned electronic device can execute the above-mentioned associated object detection method by carrying a software system corresponding to the associated object detection method. In the embodiment of the present application, the type of the above-mentioned electronic device can be a laptop computer, a computer, a server, a mobile phone, a PAD terminal, etc., which is not particularly limited in the present application.

[0056] It is understandable that the above-mentioned associated object detection method can be executed by the terminal device or the server device alone, or by the terminal device and the server device in cooperation.

[0057] For example, the above-mentioned associated object detection method may be integrated into a client. After receiving an associated object detection request, a terminal device equipped with the client may provide computing power through its own hardware environment to execute the above-mentioned associated object detection method.

[0058] For another example, the above-mentioned associated object detection method can be integrated into a system platform. After receiving an associated object detection request, a server device equipped with the system platform can provide computing power through its own hardware environment to execute the above-mentioned associated object detection method.

[0059] For example, the above-mentioned associated object detection method can be divided into two tasks: acquiring an image and performing associated object detection on the image. Among them, the acquisition task can be integrated into the client and mounted on the terminal device. The associated object detection task can be integrated into the server and mounted on the server device. The above-mentioned client can initiate an associated object detection request to the above-mentioned server device after acquiring the image. After receiving the above-mentioned request, the above-mentioned server device can execute the above-mentioned associated object detection method on the above-mentioned image in response to the above-mentioned request.

[0060] The following description is made by taking an electronic device (hereinafter referred to as device) as an example.

[0061] See also Figure 2 , Figure 2 A schematic diagram of a method for detecting associated objects shown in the present application.

[0062] like Figure 2 As shown, in Figure 2 In the method shown, it is possible to determine the associated face objects and hand objects included in the image.

[0063] The above-mentioned image refers to an image that needs to be processed. The image may include a number of detected objects. For example, in a desktop game scene, the above-mentioned image may include a number of human objects around the desktop, as well as human face objects, preset human body part objects, and human hand objects on the human body.

[0064] In some examples, when acquiring the above-mentioned image, the above-mentioned device can complete the input of the image by interacting with the user. For example, the above-mentioned device can provide the user with a window for inputting the image to be processed through its interface, so that the user can input the image. The user can complete the input of the image based on the window.

[0065] In some examples, the above-mentioned device may also be connected to an image acquisition device deployed on-site, so that the image acquired by the device may be acquired from the above-mentioned image acquisition device.

[0066] Please continue to see Figure 2 After acquiring the image, the above-mentioned device can execute the above-mentioned S102 to detect the face object, the preset body part object and the hand object contained in the image; wherein the above-mentioned preset body part object represents the preset body part in the body connection part between the face and the hand.

[0067] Among them, the body connection between the face and the hand includes the neck, shoulder, arm and wrist. The above-mentioned preset human body part object specifically represents a body part in the above-mentioned body connection. In some examples, in order to improve the correlation prediction accuracy, the above-mentioned preset human body part object can be a joint part that is relatively easy to detect by human joint detection, such as at least one of the shoulder object and the elbow object; in other examples, the above-mentioned preset human body part object can also represent the upper arm, forearm, wrist and other parts. In this way, by using the preset human body part object as an intermediary, the face object and the hand object that are far away can be associated more accurately.

[0068] In this step, the above image can be input into the object detection model for calculation, so as to detect the bounding boxes corresponding to the face object, the hand object and the preset human body part object contained in the above image. It can be understood that detecting the bounding boxes corresponding to the objects contained in the image is considered to be detecting the face object, the preset human body part object and the hand object contained in the image.

[0069] The object detection model may be a deep convolutional network model for object detection. For example, the object detection model may be a model based on an RCNN (Region Convolutional Neural Networks) network, a FAST-RCNN (Fast Region Convolutional Neural Networks) network, or a FASTER-RCNN network.

[0070] In practical applications, before using the object detection model for object detection, the model can be trained based on several image training samples that are labeled with true values ​​of face object bounding boxes, hand object bounding boxes, and preset human body part object bounding boxes until the model converges.

[0071] See also Figure 3 , Figure 3 This is a schematic diagram of the object detection process shown in this application. It should be noted that: Figure 3 The object detection process is only schematically illustrated and is not particularly limited to this application.

[0072] like Figure 3As shown, the above object detection model can be a model built based on the FASTER-RCNN network. The model can include at least a backbone network, RPN (Region Proposal Network, candidate box generation network), and RCNN (Region-based Convolutional Neural Network, region-based convolutional neural network).

[0073] Among them, the above-mentioned backbone network can perform several convolution operations on the image to obtain a feature map corresponding to the image. After obtaining the feature map, the feature map can be input into the above-mentioned RPN network to obtain several anchors. After obtaining the anchor frame, the anchor frame and the above-mentioned feature map can be input into the corresponding RCNN network for bbox (bounding boxes) regression and classification to obtain the first bounding box and the second bounding box corresponding to the face object, the hand object and the preset human body part object respectively.

[0074] It is understandable that obtaining each bounding box in this step may include obtaining position information and size information corresponding to each bounding box, etc. The position information may include vertex coordinates of the bounding box, and may also include length and width information of the bounding box.

[0075] After determining the bounding boxes corresponding to the face objects, hand objects and preset human body part objects respectively, S104 can be executed to perform correlation prediction on every two objects among the above-mentioned face objects, preset human body part objects and human hand objects, and obtain a first correlation prediction result between the above-mentioned face objects and the above-mentioned preset human body part objects, a second correlation prediction result between the above-mentioned preset human body part objects and the above-mentioned hand objects, and a third correlation prediction result between the above-mentioned face objects and the above-mentioned hand objects.

[0076] The above correlation prediction specifically refers to predicting the probability that two human body parts are mutually related objects, wherein the related objects are two human body part objects belonging to the same person.

[0077] In some examples, correlation prediction can be performed using a correlation prediction model built based on a neural network to obtain corresponding correlation prediction results.

[0078] The above-mentioned correlation prediction results (including the first correlation prediction result, the second correlation prediction result, and the third correlation prediction result, etc.) can specifically indicate the probability that the two human body parts are mutually related objects. In some examples, the correlation prediction results can be quantified by a correlation prediction score. The higher the correlation prediction score, the higher the possibility that the two human body parts corresponding to the correlation prediction score belong to the same person.

[0079] For example, in a desktop game scenario, the correlation prediction between the face object and the hand object in the image can be based on a correlation prediction model to predict the probability that any face object and any hand object detected in the image are correlated objects.

[0080] It is understandable that, under normal circumstances, the above-mentioned image may include multiple face objects, multiple preset human body part objects, and multiple hand objects. At this time, when determining the above-mentioned first correlation prediction result, any two-by-two combinations of face objects and preset human body part objects detected in the above-mentioned image can be made, and the first correlation prediction result corresponding to each obtained combination can be determined. Similarly, when determining the above-mentioned second correlation prediction result, any two-by-two combinations of preset human body part objects in the image and human hand objects in the image can be made, and the second correlation prediction result corresponding to each combination can be determined. When determining the above-mentioned third correlation prediction result, any two-by-two combinations of face objects and human hand objects in the image can be made, and the third correlation prediction result corresponding to each combination can be determined.

[0081] In some examples, when executing S104, the correlation between the face object and the preset human body part object, the correlation between the preset human body part object and the hand object, and the correlation between the face object and the hand object can be predicted respectively to obtain corresponding correlation prediction results.

[0082] It can be understood that the above three methods of determining the correlation prediction results can refer to each other, and the following is explained by taking the determination of the first correlation prediction result as an example.

[0083] For example, S1042 may be executed first to combine each detected face object with each preset human body part object in pairs to obtain a plurality of combination results.

[0084] In this step, an identifier can be created for each detected face object, each hand object, and each preset human body part object. Among them, the identifier corresponding to each part object corresponds to the part object one by one. For example, the face object identifier corresponds to the face object one by one, and the hand object identifier corresponds to the hand object one by one. It can be understood that in actual applications, the above identifiers can be part numbers. For example, face object 1, face object 2, hand object 1, preset human body part object 1, etc.

[0085] After the identification is created, each face object can be combined with each preset human body part object in sequence according to the identification (for example, the order of numbers) to obtain multiple combination results.

[0086] After obtaining the above-mentioned multiple combination results, S1044 can be executed. For each combination result, the facial object and the preset human body part object correlation prediction branch (hereinafter referred to as the first branch) is used to predict the correlation between the facial object and the preset human body part object according to the regional features and position information respectively corresponding to the facial object and the preset human body part object in the current combination result, and the first correlation prediction result corresponding to the facial object and the preset human body part object is obtained.

[0087] The first branch may be a correlation prediction model constructed based on a convolutional neural network. The correlation prediction branch may perform correlation prediction on a face object and a preset body part object to obtain a first correlation prediction result. It is understandable that the correlation prediction branch may include a fully connected layer and finally output a first correlation prediction score.

[0088] In some embodiments of the present disclosure, the fully connected layer may be a computing unit constructed based on a regression algorithm such as linear regression, least squares regression, etc. The computing unit may perform feature mapping on the regional features to obtain a corresponding correlation prediction score value. Here, the regional features represent the features of the region where the corresponding object is located in the image (e.g., the region corresponding to the bounding box of the object in the image), such as a feature map, a pixel matrix, etc. of the region where the object is located.

[0089] In practical applications, before performing correlation prediction, the first branch may be trained based on a number of image training samples having correlation annotation information between face objects and preset body part objects.

[0090] It is understandable that when training the above-mentioned face objects and preset human body part objects, several image training samples can be constructed first. Among them, when constructing the several image training samples, several original images can be obtained first. After obtaining the original images, the face objects and preset human body part objects included in the original images can be randomly combined using annotation tools to obtain multiple combination results. Then, the face objects and preset human body part objects in each combination are annotated for relevance. In some examples, if the face objects in the combination are associated with the preset human body part objects (belong to the same person), they can be annotated with 1, otherwise they can be annotated with 0. Alternatively, when annotating the original images, the information of the person objects to which each face object and each preset human body part object belongs (such as a person identification) can be annotated, thereby determining whether the face objects in the combination are associated with the preset human body part objects based on whether the information of the person objects to which they belong is consistent.

[0091] After the image training samples are constructed, the first branch can be trained based on a conventional model training method until the branch converges.

[0092] See also Figure 4 , Figure 4 This is a schematic diagram of the correlation prediction process shown in this application. Figure 4 The first branch shown may include a regional feature extraction unit and a fully connected layer. The regional feature extraction unit may obtain regional features corresponding to the face object and the preset body part object respectively based on the face object bounding box, the preset body part object bounding box and the feature map corresponding to the image.

[0093] It is understandable that in some examples, the regional features corresponding to the above-mentioned face object and the preset human body part object can be concentrated on a feature map, that is, the feature parts corresponding to the above-mentioned face object and the preset human body part object are included in a feature map. After obtaining the above-mentioned feature map, feature mapping (matrix operation) can be performed based on the feature map to obtain the corresponding correlation prediction result.

[0094] In other examples, the regional features corresponding to the above-mentioned face object and the preset human body part object can be distributed on at least two feature maps. After obtaining the feature maps corresponding to the face object and the preset human body part object, the feature maps corresponding to the face object and the preset human body part object can be input into the fully connected layer for feature splicing to obtain the spliced ​​feature map. After obtaining the spliced ​​feature map, feature mapping (matrix operation) can be performed based on the spliced ​​feature map to obtain the corresponding correlation prediction result.

[0095] The above-mentioned region feature extraction unit can be a ROI Align (Region of interest Align, region of interest feature alignment) unit or a ROI pooling (Region of interest pooling, region of interest feature pooling) unit.

[0096] The fully connected layer may be a computing unit constructed based on a regression algorithm such as linear regression, least squares regression, etc. The computing unit may perform matrix operations on regional features (feature maps, pixel matrices) to obtain corresponding correlation prediction score values.

[0097] When using the above-mentioned first branch for prediction, the combination results of each face object and the preset human body part object can be determined as the current combination result in turn, and then the regional features corresponding to the face object and the preset human body part object in the current combination result are input into the above-mentioned first branch for calculation to obtain the correlation prediction score (first correlation prediction score) between the face object and the preset human body part object in the current combination result.

[0098] In some examples, in order to improve the accuracy of the model correlation prediction, when performing the first correlation prediction score prediction, for each combination result, the first branch can be used to perform a correlation prediction between the face object and the preset human body part object in the current combination result based on the regional features and position information respectively corresponding to the face object and the preset human body part object, to obtain a first correlation prediction result corresponding to the face object and the preset human body part object.

[0099] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a correlation prediction process shown in this application. Figure 5 As shown, after the features corresponding to the face object and the features corresponding to the preset human body part object are extracted by the regional feature extraction unit, the position information (e.g., coordinates) of the face object and the position information (e.g., coordinate information) of the preset human body part object can also be feature spliced ​​(e.g., single-core convolution operation) to obtain the spliced ​​features. After obtaining the spliced ​​features, the spliced ​​features can be input into the above-mentioned fully connected layer for feature mapping (matrix operation) to obtain the first correlation prediction score value. Since when performing correlation prediction, in addition to using the features corresponding to the face object and the features corresponding to the preset human body part object, the position information corresponding to the face object bounding box and the preset human body part object bounding box are also used, so that information representing the correlation relationship between the two in position can be introduced, thereby improving the accuracy of the obtained correlation prediction results. After performing the above steps for each of the above combinations, the correlation prediction scores in the multiple above-mentioned first correlation prediction results can be obtained.

[0100] When predicting the association between the preset human body part object and the human hand object, S1046 can be executed to respectively combine the detected preset human body part objects with the human hand objects in pairs to obtain multiple combination results. S1048, for each combination result, using the preset human body part object and human hand object association prediction branch, according to the regional features and position information respectively corresponding to the preset human body part object and the human hand object in the current combination result, the preset human body part object and the human hand object are predicted to obtain a second association prediction result corresponding to the preset human body part object and the human hand object.

[0101] After the above steps are performed for each of the above combinations, relevance prediction scores in the plurality of second relevance prediction results can be obtained.

[0102] It can be understood that the step description of S1046-S1048 can refer to the step description of S1042-S1044, and will not be described in detail here.

[0103] When predicting the correlation between the face object and the hand object, the above S1042-S1044 can also be referred to, and will not be described in detail here.

[0104] It should be noted that the present application does not specifically limit the execution order of determining the first correlation prediction result, the second correlation prediction result, and the third correlation prediction result. For example, the above three correlation prediction results can be determined in sequence according to a preset order, or the first correlation prediction result, the second correlation prediction result, and the third correlation prediction result can be predicted simultaneously.

[0105] In some examples, when executing S104, the detected face object, preset body part object and hand object can be input into the correlation prediction model for calculation to obtain the above-mentioned first correlation prediction result, the above-mentioned second correlation prediction result and the above-mentioned third correlation prediction result.

[0106] The above-mentioned correlation prediction model may be a classification model including a multi-class classifier, wherein the output of the multi-class classifier may include at least the above-mentioned first correlation prediction result, the above-mentioned second correlation prediction result and the above-mentioned third correlation prediction result.

[0107] When training the above-mentioned correlation prediction model, an image training sample can be constructed first. When constructing the image training sample, the original image can be obtained first. After obtaining the original image, the image annotation tool can be used to respectively annotate the first correlation prediction result corresponding to the face object and the preset human body part object contained in each triplet included in the above-mentioned original image, the second correlation prediction result corresponding to the preset human body part object and the human hand object, and the third correlation prediction result corresponding to the face object and the human hand object, to obtain an image training sample with annotated true value. After constructing the image training sample, model training can be performed based on the constructed image training sample until the object correlation prediction model converges.

[0108] In this step, any detected face object, any preset human body part object and any human hand object can be combined to obtain multiple triplets. After obtaining the above multiple triplets, the above multiple triplets can be used as the current triplet in turn, and the face object, preset human body part object and human hand object contained in the current triplet can be input into the trained object correlation prediction model for calculation to obtain the first correlation prediction result corresponding to the face object contained in the triplet and the preset human body part object, the second correlation prediction result corresponding to the preset human body part object and the human hand object, and the third correlation prediction result corresponding to the face object and the human hand object. It should be noted that the structure of the above object correlation prediction model can refer to the structure of the aforementioned first branch, which will not be described in detail here.

[0109] Please continue to see Figure 2 After obtaining the first correlation prediction result, the second correlation prediction result and the third correlation prediction result, S106 can be continued to be executed to segment the human objects contained in the above image, and based on the areas corresponding to each of the above human objects obtained by the segmentation, determine the first human object and the second human object to which the above face object and the above preset human body part object belong respectively.

[0110] It can be understood that S106 can actually be divided into two steps, namely, S1062, segmenting the above-mentioned image, determining at least one human object contained in the image, and obtaining the area corresponding to the at least one human object; and S1064, based on the areas corresponding to each of the above-mentioned human objects obtained by segmentation, determining the first human object and the second human object to which the above-mentioned face object and the above-mentioned preset human body part object belong respectively.

[0111] In some examples, when executing S1062, the above image can be input into an instance segmentation model constructed based on a deep convolutional neural network for calculation, so as to determine the human objects included in the above image and the areas corresponding to the above human objects.

[0112] The instance segmentation model may be a model for performing instance segmentation. For example, the instance segmentation model may be a model constructed based on a FASTER-RCNN network or a MASK-RCNN (MASK Region Convolutional Neural Networks, mask-based regional convolutional neural network) network.

[0113] In practical applications, before using the instance segmentation model for instance segmentation, the model can be trained based on a number of image training samples with bounding boxes of human objects labeled until the model converges.

[0114] When constructing the above-mentioned image training sample, the original image can be obtained first. After obtaining the original image, the annotation tool can be used to determine the human objects included in the above-mentioned original image. After determining the human objects, the annotation tool can be used to respectively annotate the boundaries or contours of each human object. When annotating, different pixel values ​​can be annotated for the pixels in each human boundary box. For example, the above-mentioned original image includes human object A, human object B, and human object C. When annotating, the pixel points in the boundary box of human object A can be annotated with pixel value 1, the pixel points in the boundary box of human object B can be annotated with pixel value 2, and the pixel points in the boundary box of human object C can be annotated with pixel value 3. After the image training sample is constructed, the above-mentioned instance segmentation model can be trained based on a conventional model training method until the model converges.

[0115] See also Figure 6 , Figure 6 This is a schematic diagram of the image segmentation process shown in this application. It should be noted that: Figure 6 The image segmentation process is only schematically described, and no special limitation is imposed on this application. Figure 6 As shown, the above instance segmentation model can be a model built based on the MASK-RCNN network. The model can include at least a backbone network, RPN (Region Proposal Network), RCNN (Region-based Convolutional Neural Network), and FCNs (Fully Convolution Networks). Among them, the above backbone network can perform several convolution operations on the image to obtain a feature map corresponding to the image. After obtaining the feature map, the feature map can be input into the above RPN network to obtain several anchors. After obtaining the anchor frame, the anchor frame and the above feature map can be input into the corresponding RCNN network for bbox (bounding boxes) regression and classification to obtain the bounding boxes corresponding to each human object. After obtaining the bounding boxes corresponding to each human object, the bounding box and the above feature map can also be input into the FCNs network for semantic segmentation to obtain the region corresponding to each human object.

[0116] In some examples, the FCNs may be connected to a ROI Align unit. When the bounding box and the feature map are input into the FCNs network for semantic segmentation, the bounding box and the feature map may be first input into a ROIAlign (Region of interest Align) unit for regional feature extraction, and then the extracted regional features may be input into the FCNs network for semantic segmentation.

[0117] It is understandable that obtaining each bounding box in this step may include obtaining position information and size information corresponding to each bounding box, etc. The position information may include position coordinates determined by a rectangular coordinate system constructed based on the lower left corner of the image as the coordinate origin.

[0118] After performing instance segmentation on the above-mentioned image and obtaining an area corresponding to at least one human object, S1064 can be executed to determine the first human object and the second human object to which the above-mentioned face object and the above-mentioned preset human body part object belong respectively based on the areas corresponding to each of the above-mentioned human objects obtained by segmentation.

[0119] In some examples, when determining the first and second human body objects to which the above-mentioned face object and the above-mentioned preset human body part object belong respectively based on the area corresponding to each of the above-mentioned human body objects obtained by segmentation, the first overlapping range between the area corresponding to the face object and the area corresponding to each of the above-mentioned human body objects can be determined according to the first bounding box of the above-mentioned face object, and the second overlapping range between the area corresponding to the above-mentioned preset human body part object and the area corresponding to each of the above-mentioned human body objects can be determined according to the second bounding box of the above-mentioned preset human body part object. After determining the above-mentioned first overlapping range and the second overlapping range, the first and second human body objects to which the above-mentioned face object and the above-mentioned preset human body part object belong respectively can be determined according to the above-mentioned first overlapping range and the second overlapping range.

[0120] Usually, for the same human object, the first bounding box of the face object and the second bounding box of the preset human body part object are located within the human body bounding box. Therefore, the size of the first overlapping range can represent the possibility that the face object belongs to a certain human object, and the size of the second overlapping range can represent the possibility that the preset human body part object belongs to a certain human object. Taking a face object as an example, if the first overlap range between a face object A1 and a human body object B is larger than the first overlap range between a face object A2 and human body object B, it means that the possibility that the face object A1 belongs to the human body object B is greater than the possibility that the face object A2 belongs to the human body object B; or, if the first overlap range between a face object A1 and a human body object B exceeds a predetermined interval (for example, the area of ​​the overlap range exceeds 80% of the corresponding area of ​​the face object), it means that the possibility or probability value of the face object A1 belonging to the human body object B is higher. At this time, it can be directly determined that the face object A1 belongs to the human body object B, or the human body object B is added to the candidate human body object set to which the face object A1 belongs, and further according to the ranking of the probability values ​​of the face object A1 belonging to each human body object in the corresponding candidate human body object set, the first human body object to which the face object belongs is determined.

[0121] It can be understood that if, in the process of determining the first human body object and the second human body object to which the above-mentioned face object and the above-mentioned preset human body part object belong respectively according to the first overlap range and the above-mentioned second overlap range, there is a situation where at least two face objects belong to the same human body object, or when the preset human body part object is an elbow object or a shoulder object, there is a situation where at least three preset human body part objects belong to the same human body object, then the face objects and the preset human body part objects belonging to the same human body object can be sorted and screened in order of their overlap ranges from large to small according to the constraint condition that the same human body object is associated with at most one face object, two elbow objects, and two shoulder objects, so as to reduce the possibility that the determination result of the human body object to which the face object and the preset human body object belong does not conform to the actual situation. Therefore, in this example, by determining the first overlap range between the area corresponding to the above-mentioned face object and the area corresponding to each of the above-mentioned human body objects and the second overlap range between the area corresponding to the above-mentioned preset human body part object and the area corresponding to each of the above-mentioned human body objects, the first human body object and the second human body object to which the above-mentioned face object and the above-mentioned preset human body part object belong respectively can be determined.

[0122] In some examples, when determining the first human body object and the second human body object to which the face object and the preset human body part object belong respectively according to the first overlap range and the second overlap range, it is possible to determine a first target area with the largest first overlap range with the area corresponding to the face object among the areas corresponding to the human body objects respectively; and determine a second target area with the largest second overlap range with the area corresponding to the preset human body part object among the areas corresponding to the human body objects respectively. After determining the first target area and the second target area, determine the human body object corresponding to the first target area as the first human body object to which the face object belongs, and determine the human body object corresponding to the second target area as the second human body object to which the preset human body part object belongs.

[0123] Since the first target area is the area with the largest first overlap range with the area corresponding to the facial object, and the second target area is the area with the largest second overlap range with the area corresponding to the preset human body part object, the first human body object to which the facial object is most likely to belong and the second human body object to which the preset human body part object is most likely to belong can be determined in this example, thereby ensuring the accuracy of associated object detection. In some examples, when determining the first target area, the pixels contained in the area corresponding to the facial object can be grouped according to the pixel value size to obtain a plurality of classification combinations. It can be understood that since different pixel values ​​represent different areas corresponding to human objects, after classifying the pixels, each classification combination can represent an area corresponding to the human body object.

[0124] After classifying the pixels, the number of pixels contained in each classification combination can be counted, and the human object area corresponding to the classification combination containing the highest number of pixels can be determined as the above-mentioned first target area. For example, assuming that the area corresponding to the face object includes a total of 100 pixels, of which 80 correspond to the area corresponding to human object A, 15 correspond to the area corresponding to human object B, and 5 correspond to the area corresponding to human object C. At this time, the area corresponding to human object A can be considered as the above-mentioned first target area. It should be noted that the method for determining the above-mentioned second target area can refer to the above-mentioned method for determining the first target area, which will not be described in detail here.

[0125] In some examples, when determining the first target region and the second target region, an IoU (Intersection over Union) algorithm or the like may be used for determination.

[0126] In some examples, when determining the first human body object and the second human body object to which the facial object and the preset human body part object belong respectively based on the areas corresponding to the human body objects obtained by segmentation, after determining the first overlapping range and the second overlapping range, the merged range of the facial object and the human body object, as well as the merged range of the preset human body part object and the human body object are determined respectively according to the area of ​​the facial object and the area of ​​the preset human body part object.

[0127] After determining the above merging range, based on the IoU algorithm formula, the first overlapping range between the above face object and the above body object is divided by the corresponding merging range of the above two to obtain the first IoU value set corresponding to the above face object. Similarly, the second IoU value set corresponding to the above preset body part object can be obtained.

[0128] After determining the first IoU value set and the second IoU value set, the third IoU value with the largest value in the first IoU value set and the fourth IoU value with the largest value in the second IoU value set can be determined. Afterwards, the human body object corresponding to the third IoU value and the human body object corresponding to the fourth IoU value are respectively determined as the first human body object to which the face object belongs and the second human body object to which the preset human body part object belongs.

[0129] The size of the above IoU value can represent the possibility that the face object or the preset human body part object belongs to a certain human body object. For example, if the IoU value between a face object and a human body object is large, it means that the face object is likely to belong to the human body object.

[0130] Since the third IoU value is the maximum value in the first IoU value set and the fourth IoU value is the maximum value in the second IoU value set, it is possible to determine, in this example, the first human object to which the face object is most likely to belong and the second human object to which the preset human body part object is most likely to belong, thereby ensuring the accuracy of associated object detection.

[0131] In some examples, when executing S1064, it may be impossible to determine the first human body object to which the face object belongs or the second human body object to which the preset human body part object belongs. If such a situation occurs, the correlation prediction result corresponding to the face object or the preset human body object may not be adjusted.

[0132] For example, when determining the first human object to which a facial object belongs, there may be several human object areas that overlap with the area corresponding to the facial object, and the first overlapping ranges between the areas of the facial object are roughly the same in size. At this time, it can be considered that the first human object to which the facial object belongs cannot be determined, and the association prediction result corresponding to the facial object may not be adjusted.

[0133] Please continue to see Figure 2 After determining the first human body object and the second human body object to which the face object and the preset human body part object contained in the above image belong respectively, S108 can be continued to be executed, and the first correlation prediction result can be adjusted based on the first human body object and the second human body object, and the associated objects in the above image can be determined according to the adjusted first correlation prediction result, the second correlation prediction result and the third correlation prediction result.

[0134] It can be understood that S108 can actually be divided into two steps, namely, S1082, adjusting the above-mentioned first correlation prediction result based on the first human object and the second human object; and S1084, determining the associated object in the above-mentioned image according to the adjusted above-mentioned first correlation prediction result, the above-mentioned second correlation prediction result and the above-mentioned third correlation prediction result.

[0135] In some examples, when executing S1082, the first human body object to which the face object belongs can be matched with the second human body object to which the preset human body part object belongs to obtain a matching result. After obtaining the matching result, the first correlation prediction result can be adjusted based on the matching result.

[0136] If the first human body object to which the above-mentioned face object belongs matches the second human body object to which the above-mentioned preset human body part object belongs, it can be explained that the above-mentioned face object and the above-mentioned preset human body part object are likely to belong to the same person. Therefore, in this example, based on the above-mentioned matching result, the above-mentioned first correlation prediction result is adjusted. By adjusting the above-mentioned first correlation prediction result, the possibility of determining that the face object corresponding to the first correlation prediction result and the preset human body part object belong to the same person is improved, thereby improving the accuracy of associated object detection.

[0137] It is understandable that, since the image includes a combination of multiple pairs of face objects and preset human body part objects, multiple first correlation prediction results can usually be determined when executing S104. At this time, when executing S1082, the above multiple first correlation prediction results can be used as the current first correlation prediction results in sequence, and the following steps are performed: matching the first human body object with the second human body object to obtain a matching result. Afterwards, the current first correlation prediction result can be adjusted based on the above matching results.

[0138] In some examples, the first correlation prediction result can be adjusted according to the following ideas. The correlation prediction result can include a correlation prediction score, and the higher the correlation prediction score, the stronger the correlation between the two. When the first human body object matches the second human body object, the correlation prediction score in the first correlation prediction result is increased; and / or, when the first human body object is inconsistent with the second human body object, the correlation prediction score in the first correlation prediction result is reduced.

[0139] In this example, since the correlation prediction score in the first correlation prediction result can be increased when the first human object matches the second human object, and / or the correlation score in the first correlation prediction result can be reduced when the first human object matches the second human object. Therefore, the correlation prediction score between the face object belonging to the same person and the preset human body part object can be increased, and / or the correlation prediction score between the face object not belonging to the same person and the preset human body part object can be reduced, thereby improving the accuracy of associated object detection.

[0140] It should be noted that the present application does not specifically limit the method of increasing or decreasing the correlation prediction score in the first correlation prediction result. For example, when increasing or decreasing the correlation prediction score in the first correlation prediction result, the preset value may be increased or decreased based on the correlation prediction score in the original first correlation prediction result.

[0141] In some examples, the segmentation result obtained by performing instance segmentation on the image includes not only the areas corresponding to the human objects contained in the above image, but also a confidence score indicating the accuracy of the instance segmentation result. At this time, when increasing the correlation prediction score in the first correlation prediction result between the face object and the preset human body part object, the correlation prediction score in the first correlation prediction result can be increased to the confidence score corresponding to the human body object to which the face object belongs. When reducing the correlation prediction score in the first correlation prediction result, the correlation prediction score in the original first correlation prediction result can be halved.

[0142] In some examples, when executing S1064, it may be impossible to determine the first human body object to which the face object belongs or the second human body object to which the preset human body part object belongs. If such a situation occurs, the correlation prediction result corresponding to the face object or the preset human body object may not be adjusted.

[0143] Please continue to see Figure 2 After adjusting the first correlation prediction result, step S1084 may be continued to determine the associated object in the image according to the adjusted first correlation prediction result, the second correlation prediction result and the third correlation prediction result.

[0144] In some examples, when the above image only includes a human face object, a preset human body part object and a human hand object, when determining the associated objects in the above image, the association prediction score in the above first association prediction result, the association prediction score in the above second association prediction result and the association prediction score in the above third association prediction result can be added to obtain an addition result. After obtaining the above addition result, it can be determined whether the above addition result reaches the preset standard value. If the above addition result reaches the above preset standard value, it can be considered that the human face object and the human hand object included in the figure are associated objects. In some embodiments of the present disclosure, the above preset standard value is specifically an empirical threshold that can be set according to the actual situation. For example, the preset standard value can be 0.95.

[0145] When the above image includes multiple face objects, preset human body part objects and human hand objects, when determining the associated objects in the above image, the detected face objects, the preset human body part objects and the human hand objects can be combined to generate at least one triple, wherein each triple includes a face object, a preset human body part object and a hand object. After obtaining multiple triples, the associated objects in the above image can be determined.

[0146] In some embodiments of the present disclosure, the fourth relevance prediction result corresponding to each triple may be determined first according to the adjusted first relevance prediction result, the second relevance prediction result and the third relevance prediction result.

[0147] In some examples, the fourth correlation prediction result may include a correlation prediction score. When determining the correlation prediction scores in the correlation prediction results corresponding to the above-mentioned multiple triples, each triple in the above-mentioned multiple triples may be determined as the current triple in turn. Then, the sum of the correlation prediction scores in the adjusted first correlation prediction result corresponding to the face object and the preset human body part object contained in the current triple, the correlation prediction scores in the second correlation prediction result corresponding to the preset human body part object and the human hand object, and the correlation prediction scores in the third correlation prediction result corresponding to the face object and the human hand object is determined as the correlation prediction score in the correlation prediction result corresponding to the current triple.

[0148] It should be noted that the present application does not specifically limit the specific method of determining the correlation prediction scores in the correlation prediction results corresponding to the above-mentioned multiple triples respectively according to the adjusted correlation prediction scores in the above-mentioned first correlation prediction results, the correlation prediction scores in the above-mentioned second correlation prediction results, and the correlation prediction scores in the above-mentioned third correlation prediction results. For example, the correlation prediction scores in the correlation prediction results corresponding to the triples can also be determined by multiplying the above-mentioned three types of scores, or by weighted summing or averaging.

[0149] After obtaining the correlation prediction results corresponding to each of the plurality of triples, the associated objects in the image may be determined based on the correlation prediction results corresponding to each triple.

[0150] The fourth correlation prediction result corresponding to each of the above triples can represent the possibility that the face object and the hand object contained in the triple belong to the same person. For example, when the fourth correlation prediction result is represented by the correlation prediction score in the correlation prediction result, if the correlation prediction score in the correlation prediction result is high, it means that the face object and the hand object in the triple corresponding to the correlation prediction result are likely to belong to the same person.

[0151] Therefore, in this example, by determining the associated objects in the above image based on the correlation prediction results corresponding to each triplet, the face objects and hand objects in the closely associated triplet can be determined as associated objects, thereby improving the accuracy of associated object detection.

[0152] In some examples, when determining the associated objects in the above image based on the fourth association prediction results corresponding to each triplet, the face objects and hand objects included in the triplet with a high association prediction score in the corresponding fourth association prediction result can be preferentially determined as associated face objects and hand objects.

[0153] A high correlation prediction score in the correlation prediction result indicates that the correlation between the face object and the hand object in the triplet is strong. Therefore, in some examples, it can be judged whether the face object and the hand object in the triplet are related objects in order from high to low according to the correlation prediction scores in the correlation prediction result.

[0154] In actual situations, usually a face object can only correspond to two hand objects at most, and a hand object can only correspond to one face object at most. In order to meet the above actual situations, in some examples, when the face objects and hand objects included in the triples with higher correlation prediction scores in the corresponding correlation prediction results are preferentially determined as associated face objects and hand objects, the current triples can be determined in turn according to the sorting of the correlation prediction scores in the above correlation prediction results corresponding to each triple from high to low, and the following operations are performed on each current triple: based on the determined associated objects, determine whether the number of associated hand objects that are associated objects with the face objects included in the current triple has reached a first preset threshold, and determine whether the number of associated face objects that are associated objects with the hand objects included in the current triple has reached a second preset threshold.

[0155] The first preset threshold is specifically an empirical threshold that can be set according to actual circumstances. For example, the first preset threshold can be 2.

[0156] The second preset threshold is specifically an empirical threshold that can be set according to actual circumstances. For example, the second preset threshold can be 1.

[0157] In some examples, a counter may be maintained for each face object and each hand object, and whenever an associated hand object is determined to be an associated object with any face object, the value on the counter corresponding to the associated face object may be increased by 1. At this time, when determining whether the number of associated hand objects that are associated objects with the face object included in the current triplet has reached a first preset threshold based on the determined associated objects, it is possible to determine whether the number of associated hand objects that are associated objects with the face object has reached the first preset threshold by determining whether the count value of the counter corresponding to the face object has reached the first preset threshold. When determining whether the number of associated face objects that are associated objects with the hand object included in the current triplet has reached a second preset threshold, it is possible to determine whether the number of associated face objects that are associated objects with the hand object has reached the second preset threshold by determining whether the count value of the counter corresponding to the hand object has reached the second preset threshold.

[0158] If the number of associated hand objects associated with the face objects included in the current triplet does not reach the first preset threshold, and the number of associated face objects associated with the hand objects included in the current triplet does not reach the second preset threshold, the face objects and hand objects included in the current triplet are determined as associated objects in the above image.

[0159] When determining the associated objects, if the number of associated hand objects associated with the face object included in the current triplet does not reach the first preset threshold, and the number of associated face objects associated with the hand object included in the current triplet does not reach the second preset threshold, the face object and the hand object in the current triplet can be determined as associated objects. Therefore, it is possible to avoid the situation where a face object is associated with more than two hand objects and a hand object is associated with more than one face object.

[0160] In some examples, after determining the associated object in the image, the detection result of the associated object in the image can be output. In some examples, the bounding box of the face object and the hand object containing the above-mentioned associated object indication can be output on the image output device (such as a display). By displaying the bounding box containing the above-mentioned associated object, the observer can conveniently and intuitively determine the associated object in the image displayed on the image output device, thereby facilitating further manual verification of the detection result of the associated object.

[0161] The above is an introduction to the scheme for determining associated objects in an image shown in this application. The following introduces the training methods of each model used in the scheme.

[0162] Please continue to see Figure 2 .exist Figure 2In the illustrated scheme, an object detection model based on a neural network can be used to detect face objects, preset human body part objects, and human hand objects contained in an image. A correlation prediction model based on a neural network can be used to predict the correlation between every two of the above face objects, preset human body part objects, and human hand objects. An instance segmentation model based on a neural network can be used to segment the human body objects contained in the above image. It is understandable that the above object detection model, the above instance segmentation model, and the above correlation prediction model can share the same backbone network.

[0163] In some examples, training sample sets may be constructed for the object detection model, the instance segmentation model, and the relevance prediction model, respectively, and the object detection model, the instance segmentation model, and the relevance prediction model may be trained based on the constructed training sample sets.

[0164] In some examples, in order to improve the accuracy of related object detection, each model can be trained in a segmented training manner. The first segment is the training of the object detection model and the instance segmentation model; the second segment is the joint training of the object detection model, the instance segmentation model and the relatedness prediction model.

[0165] See also Figure 7 , Figure 7 A flowchart of the model training method according to an embodiment of the present application.

[0166] like Figure 7 As shown, the method includes: S702, training the above-mentioned object detection model based on a first training sample set; wherein the above-mentioned first training sample set includes image training samples with first annotation information; the above-mentioned first annotation information includes the bounding box of the face object, the above-mentioned preset body part object and the hand object.

[0167] When executing this step, the original image can be annotated with true values ​​by manual annotation or machine-assisted annotation. For example, after obtaining the original image, the image annotation tool can be used to annotate the face object bounding box, the hand object bounding box, and the preset human body part object (such as the elbow) bounding box included in the original image to obtain several image training samples. It should be noted that when encoding the image training samples, one-hot encoding and other methods can be used for encoding, and this application does not limit the specific encoding method.

[0168] After the first training sample set is determined, the object detection model may be trained based on a preset loss function until the model converges.

[0169] S704, training the instance segmentation model based on a second training sample set; wherein the second training sample set includes image training samples with second annotation information; and the second annotation information includes a bounding box of a human object.

[0170] When constructing the above-mentioned image training sample, the original image can be obtained first. After obtaining the original image, the annotation tool can be used to determine the human objects included in the above-mentioned original image. After determining the human objects, the annotation tool can be used to respectively annotate the bounding boxes corresponding to the human objects. When annotating, different pixel values ​​can be annotated for the pixels in each bounding box. For example, the above-mentioned original image includes human object A, human object B, and human object C. When annotating, the pixel points in the bounding box of human object A can be annotated with pixel value 1, the pixel points in the bounding box of human object B can be annotated with pixel value 2, and the pixel points in the bounding box of human object C can be annotated with pixel value 3. After determining the second training sample set, the above-mentioned instance segmentation model can be trained based on a conventional model training method until the model converges.

[0171] S706, jointly train the above-mentioned object detection model, the above-mentioned instance segmentation model and the above-mentioned correlation prediction model based on a third training sample set; wherein the above-mentioned third training sample set includes image training samples with third annotation information; the above-mentioned third annotation information includes bounding boxes corresponding to face objects, preset body part objects, hand objects and human body objects respectively, as well as correlation annotation information between face objects and preset body part objects, correlation annotation information between preset body part objects and hand objects, and correlation annotation information between face objects and hand objects.

[0172] When executing this step, the original image can be annotated with true values ​​by manual annotation or machine-assisted annotation. For example, after obtaining the original image, on the one hand, the image annotation tool can be used to annotate the bounding box of the face object, the bounding box of the hand object, and the bounding box of the preset human body part object (such as the elbow) included in the original image. On the other hand, the image annotation tool can be used to determine the human body objects included in the above-mentioned original image. After determining each human body object, the annotation tool can be used to annotate the bounding box corresponding to each human body object respectively. On the other hand, the annotation tool can be used to arbitrarily combine the face object and the preset human body part object included in the original image, arbitrarily combine the face object and the hand object included in the original image, and arbitrarily combine the preset human body part object and the hand object included in the original image to obtain multiple combination results. Then, the correlation result is annotated for the two objects in each combination. In some examples, if the two objects in the combination belong to the same person, they are annotated with 1, otherwise they are annotated with 0.

[0173] After determining the third training sample set, the joint learning loss function can be determined based on the loss functions corresponding to the above-mentioned object detection model, the above-mentioned instance segmentation model, and the above-mentioned relevance prediction model. In some examples, the loss functions corresponding to the above-mentioned object detection model, the above-mentioned instance segmentation model, and the above-mentioned relevance prediction model can be added to obtain the above-mentioned joint learning loss function. It should be noted that in the present application, hyperparameters such as regularization terms can also be added to the above-mentioned joint learning loss function. The types of added hyperparameters are not particularly limited here.

[0174] After obtaining the joint training loss function, the object detection model, the instance segmentation model and the relevance prediction model can be jointly trained based on the joint learning loss function and the third training sample set until the object detection model, the instance segmentation model and the relevance prediction model converge.

[0175] When training the model, hyperparameters such as learning rate, number of training cycles, etc. may be specified first. After the hyperparameters are determined, supervised training may be performed on each model based on the image training samples labeled with true values.

[0176] In a supervised training process, forward propagation can be performed to obtain the calculation results of each model output. After obtaining the calculation results of each model output, the error of the above calculation results, that is, the difference between the above calculation results and the true value, can be evaluated based on the constructed joint learning loss function. After obtaining the error, the stochastic gradient descent method can be used to determine the descent gradient. After determining the descent gradient, the model parameters corresponding to each of the above models can be updated based on back propagation. The above process can then be repeated based on each model after the updated parameters until each of the above models converge. It should be noted that the conditions for the convergence of the above models can be such as reaching a preset number of training times, or the change in the joint learning loss function obtained after M consecutive (M is a positive integer greater than 1) forward propagations is less than a certain threshold value. This application does not specifically limit the conditions for model convergence.

[0177] Since a supervised joint training method is used in the above model training, each model can be trained simultaneously, so that the models can both constrain and promote each other during the training process, thereby improving the convergence efficiency of each model on the one hand; on the other hand, it promotes the backbone network shared by each model to extract features that are more beneficial to the detection of related objects, thereby improving the accuracy of related object detection.

[0178] This application also proposes a detection device for associated objects in an image. Figure 8 , Figure 8 This is a structural diagram of a device for detecting associated objects in an image shown in the present application.

[0179] like Figure 8 As shown, the above-mentioned device 80 includes: an object detection module 81, which is used to detect a face object, a preset human body part object and a human hand object contained in an image; wherein the above-mentioned preset human body part object represents a preset human body part in the body connection part between the face and the hand;

[0180] A correlation prediction module 82 is used to perform correlation prediction on every two objects among the above-mentioned face objects, preset human body part objects and human hand objects, and obtain a first correlation prediction result between the above-mentioned face objects and the above-mentioned preset human body part objects, a second correlation prediction result between the above-mentioned preset human body part objects and the above-mentioned human hand objects, and a third correlation prediction result between the above-mentioned face objects and the above-mentioned human hand objects;

[0181] an attribution relationship determination module 83, configured to segment the human objects contained in the image, and determine the first human object and the second human object to which the face object and the preset human body part object belong respectively based on the regions corresponding to the human objects obtained by segmentation; and

[0182] The associated object determination module 84 is used to adjust the above-mentioned first correlation prediction result based on the first human object and the second human object, and determine the associated object in the above-mentioned image according to the adjusted above-mentioned first correlation prediction result, the above-mentioned second correlation prediction result and the above-mentioned third correlation prediction result.

[0183] In some of the illustrated embodiments, the object detection module 81 is specifically used to: detect the first bounding box and the second bounding box corresponding to the facial object and the preset human body part object in the image respectively; the attribution relationship determination module 83 includes: an overlap range determination module, used to determine the first overlap range between the area corresponding to the facial object and the area corresponding to each of the human body objects according to the first bounding box, and to determine the second overlap range between the area corresponding to the preset human body part object and the area corresponding to each of the human body objects according to the second bounding box; and an attribution relationship determination submodule, used to determine the first human body object and the second human body object to which the facial object and the preset human body part object belong respectively according to the first overlap range and the second overlap range.

[0184] In some of the illustrated embodiments, the above-mentioned attribution relationship determination submodule is specifically used to: determine a first target area with the largest first overlap range with the area corresponding to the above-mentioned facial object among the areas corresponding to the above-mentioned human objects respectively; determine a second target area with the largest second overlap range with the area corresponding to the above-mentioned preset human body part object among the areas corresponding to the above-mentioned human objects respectively; and determine that the human body object corresponding to the above-mentioned first target area is the first human body object to which the facial object belongs; and determine that the human body object corresponding to the above-mentioned second target area is the second human body object to which the above-mentioned preset human body part object belongs.

[0185] In some of the illustrated embodiments, the associated object determination module 84 is specifically used to: match the first human object with the second human object to obtain a matching result; and adjust the first association prediction result based on the matching result.

[0186] In some of the illustrated embodiments, the associated object determination module 84 is specifically used to: when the first human body object matches the second human body object, increase the association prediction score in the first association prediction result; and / or, when the first human body object does not match the second human body object, reduce the association score in the first association prediction result.

[0187] In some of the illustrated embodiments, the device 80 further includes: a combination module, which is used to combine the detected face object, the preset human body part object and the hand object to generate at least one triple, wherein each triple includes a face object, a preset human body part object and a hand object; the correlation prediction module 82 is specifically used to: perform correlation prediction on every two objects in each triple; the associated object determination module 84 includes: a triple correlation relationship prediction module, which is used to determine the fourth correlation prediction result corresponding to each triple based on the adjusted first correlation prediction result, the second correlation prediction result and the third correlation prediction result; and an associated object determination submodule, which is used to determine the associated objects in the image based on the fourth correlation prediction result corresponding to each triple.

[0188] In some of the illustrated embodiments, the associated object determination submodule is specifically used to: determine the current triplet for each triplet in turn according to the order of the association prediction scores in the fourth association prediction results corresponding to each triplet from high to low, and execute: based on the determined associated objects, determine whether the number of associated hand objects that are mutually associated with the face objects included in the current triplet reaches a first preset threshold, and determine whether the number of associated face objects that are mutually associated with the hand objects included in the current triplet reaches a second preset threshold; if the number of associated hand objects does not reach the first preset threshold, and the number of associated face objects does not reach the second preset threshold, then determine the face object and hand object included in the current triplet as associated objects in the above image.

[0189] In some of the illustrated embodiments, the preset human body part object includes at least one of a shoulder object and an elbow object.

[0190] In some of the illustrated embodiments, the apparatus 80 further includes: an output module, configured to output a detection result of the associated object in the image.

[0191] In some of the illustrated embodiments, the object detection module 81 is specifically used to: use an object detection model constructed based on a neural network to detect facial objects, preset human body part objects and human hand objects contained in the image; the association prediction module 82 is specifically used to: use an association prediction model constructed based on a neural network to perform association prediction on every two of the facial objects, preset human body part objects and human hand objects; the attribution relationship determination module 83 is specifically used to: use an instance segmentation model constructed based on a neural network to segment the human body objects contained in the image.

[0192] In some of the illustrated embodiments, the apparatus 80 further includes: a first training module for training the object detection model based on a first training sample set; wherein the first training sample set includes image training samples with first annotation information; the first annotation information includes bounding boxes of face objects, preset body part objects, and hand objects; a second training module for training the instance segmentation model based on a second training sample set; wherein the second training sample set includes image training samples with second annotation information; the second annotation information includes bounding boxes of human objects; and a joint training module for jointly training the object detection model, the instance segmentation model, and the correlation prediction model based on a third training sample set; wherein the third training sample set includes image training samples with third annotation information; the third annotation information includes bounding boxes corresponding to face objects, preset body part objects, hand objects, and human objects, respectively, as well as correlation annotation information between face objects and preset body part objects, correlation annotation information between preset body part objects and hand objects, and correlation annotation information between face objects and hand objects.

[0193] The embodiments of the apparatus for detecting associated objects in an image shown in the present application can be applied to electronic devices. Accordingly, the present application discloses an electronic device, which may include: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to call the executable instructions stored in the memory to implement the method for detecting associated objects in an image as shown in any of the above embodiments.

[0194] See also Fig. 9 , Fig. 9 FIG. 1 is a hardware structure diagram of an electronic device shown in this application. Fig. 9 As shown, the electronic device may include a processor for executing instructions, a network interface for network connection, a memory for storing operating data for the processor, and a non-volatile memory for storing instructions corresponding to the image processing device. The image processing device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From a hardware perspective, in addition to Fig. 9 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in the embodiment where the device is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail. It is understandable that in order to increase the processing speed, the corresponding instructions of the image processing device can also be directly stored in the memory, which is not limited here.

[0195] The present application proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the method for detecting associated objects in an image as shown in any of the above embodiments.

[0196] The present application also proposes a computer program product, including a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the method for detecting associated objects in an image as shown in any of the above embodiments.

[0197] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0199] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0200] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0201] The embodiments of the subject matter and functional operations described in this application can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that can include the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, that is, one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer-readable storage medium can be a machine-readable storage device, a machine-readable storage medium, a random or serial access memory device, or a combination of one or more of them.

[0202] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The above processes and logic flows can also be performed by special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special-purpose logic circuits.

[0203] A computer suitable for executing a computer program may include, for example, a general and / or special purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0204] Computer readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0205] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work as mentioned above in some combinations and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0206] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0207] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0208] The above is only a preferred embodiment of one or more embodiments of the present application, and is not intended to limit one or more embodiments of the present application. Any modification, equivalent replacement, improvement, etc. made without departing from the spirit and principle of one or more embodiments of the present application are included in the scope of protection defined by the attached claims.

Claims

1. A method for detecting associated objects in an image, characterized in that: The method comprises: Detecting multiple face objects, multiple preset human body part objects, and multiple human hand objects contained in an image; wherein the preset human body part objects represent preset human body parts in the body connection portion between a face and a hand; Performing correlation prediction on every two of the face object, the preset human body part object and the human hand object to obtain a first correlation prediction result between the face object and the preset human body part object, a second correlation prediction result between the preset human body part object and the human hand object, and a third correlation prediction result between the face object and the human hand object; Segmenting the human body object contained in the image, and Determine, based on the regions corresponding to the human body objects obtained by segmentation, the first human body object and the second human body object to which the face object and the preset human body part object belong respectively; Adjusting the first correlation prediction result based on the first human object and the second human object includes: Matching the first human object with the second human object to obtain a matching result, Based on the matching result, adjusting the first correlation prediction result; Determine the associated object in the image according to the adjusted first association prediction result, the second association prediction result and the third association prediction result.

2. The method according to claim 1, characterized in that The detection image contains a plurality of face objects, a plurality of preset human body part objects, and a plurality of human hand objects, including: Detecting a first bounding box and a second bounding box corresponding to a face object and a preset body part object in an image, respectively; The determining, based on the area corresponding to each of the human body objects obtained by segmentation, the first human body object and the second human body object to which the face object and the preset human body part object belong respectively, comprises: Determine, according to the first bounding box, a first overlapping range between the area corresponding to the face object and the areas corresponding to each of the human body objects; Determine, according to the second bounding box, a second overlapping range between the area corresponding to the preset human body part object and the area corresponding to each of the human body objects; The first human body object and the second human body object to which the human face object and the preset human body part object belong respectively are determined according to the first overlapping range and the second overlapping range.

3. The method according to claim 2, characterized in that The method of determining the first human body object and the second human body object to which the face object and the preset human body part object belong respectively according to the first overlapping range and the second overlapping range comprises: Determine a first target area having the largest first overlap range with the area corresponding to the face object among the areas corresponding to the human body objects respectively; Determine a second target area having the largest second overlap range with the area corresponding to the preset human body part object among the areas corresponding to the human body objects respectively; Determining that the human body object corresponding to the first target area is the first human body object to which the human face object belongs; and It is determined that the human body object corresponding to the second target area is the second human body object to which the preset human body part object belongs.

4. The method according to claim 1, characterized in that: The adjusting the first relevance prediction result based on the matching result includes: In the case where the first human object matches the second human object, increasing the correlation prediction score in the first correlation prediction result; and / or, In the case where the first human object does not match the second human object, the correlation prediction score in the first correlation prediction result is reduced.

5. The method according to claim 1, characterized in that The method further comprises: Combining the detected face object, the preset human body part object and the human hand object to generate at least one triple, wherein each triple includes a face object, a human body part object and a hand object; The predicting of the association between each two objects among the face object, the preset body part object and the hand object includes: Make association predictions for every two objects in each triple; The determining the associated object in the image according to the adjusted first association prediction result, the second association prediction result and the third association prediction result includes: Determine a fourth relevance prediction result corresponding to each triple according to the adjusted first relevance prediction result, the second relevance prediction result, and the third relevance prediction result; The associated objects in the image are determined based on the fourth association prediction results corresponding to each triple.

6. The method according to claim 5, characterized in that The determining the associated object in the image based on the fourth association prediction result corresponding to each triplet includes: According to the order of the relevance prediction scores in the fourth relevance prediction results corresponding to the triples from high to low, the triples are determined as current triples in turn, and the following is performed: Based on the determined associated objects, determining whether the number of associated hand objects that are associated with the face object included in the current triplet has reached a first preset threshold, and determining whether the number of associated face objects that are associated with the hand object included in the current triplet has reached a second preset threshold; If the number of associated hand objects does not reach the first preset threshold, and the number of associated face objects does not reach the second preset threshold, the face objects and hand objects included in the current triplet are determined as associated objects in the image.

7. The method according to claim 1, characterized in that The preset human body part object includes at least one of a shoulder object and an elbow object.

8. The method according to claim 1, characterized in that: The method further comprises: Output the detection result of the associated object in the image.

9. The method according to claim 1, characterized in that: The detection image contains a plurality of face objects, a plurality of preset human body part objects, and a plurality of human hand objects, including: An object detection model based on a neural network is used to detect multiple face objects, multiple preset human body parts objects, and multiple human hand objects contained in the image; The predicting of the association between each two objects among the face object, the preset body part object and the hand object includes: Using a correlation prediction model built based on a neural network, a correlation prediction is performed on every two objects among the face object, the preset body part object and the hand object; The step of segmenting the human body object contained in the image comprises: An instance segmentation model constructed based on a neural network is used to segment the human body objects contained in the image.

10. The method according to claim 9, characterized in that The method further comprises: The object detection model is trained based on a first training sample set; wherein the first training sample set includes image training samples with first annotation information; the first annotation information includes a bounding box of a face object, the preset human body part object, and a hand object; Training the instance segmentation model based on a second training sample set; wherein the second training sample set includes image training samples with second annotation information; the second annotation information includes a bounding box of a human object; The object detection model, the instance segmentation model and the association prediction model are jointly trained based on a third training sample set; wherein the third training sample set includes image training samples with third annotation information; the third annotation information includes bounding boxes corresponding to face objects, preset human body part objects, hand objects and human body objects respectively, as well as association annotation information between face objects and preset human body part objects, association annotation information between preset human body part objects and hand objects, and association annotation information between face objects and hand objects.

11. A device for detecting associated objects in an image, characterized in that: The device comprises: An object detection module, used to detect multiple face objects, multiple preset human body part objects and multiple human hand objects contained in the image; wherein the preset human body part objects represent preset human body parts in the body connection part between the human face and the human hand; A correlation prediction module is used to perform correlation prediction on every two objects among the face object, the preset human body part object and the human hand object, to obtain a first correlation prediction result between the face object and the preset human body part object, a second correlation prediction result between the preset human body part object and the human hand object, and a third correlation prediction result between the face object and the human hand object; an attribution relationship determination module, configured to segment the human objects contained in the image, and determine the first human object and the second human object to which the face object and the preset human body part object belong respectively based on the regions corresponding to the human objects obtained by segmentation; An associated object determination module is used to adjust the first association prediction result based on the first human object and the second human object to which the face object and the preset human body part object belong respectively, including: matching the first human object with the second human object to obtain a matching result, and adjusting the first association prediction result based on the matching result; and determining the associated object in the image according to the adjusted first association prediction result, the second association prediction result and the third association prediction result.

12. An electronic device, characterized in that: The device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to call the executable instructions stored in the memory to implement the method for detecting associated objects in an image according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to be executed by a processor to implement the method for detecting associated objects in an image according to any one of claims 1 to 10.

14. A computer program product, characterized in that The method comprises a computer readable code, and when the computer readable code is executed in an electronic device, a processor in the electronic device executes the method according to any one of claims 1 to 10.

Images