Model training method and device, computer device, and storage medium

Abstract

Embodiments of the present application disclose a model training method and device, computer equipment and a storage medium, wherein the method comprises: obtaining a sample pair and a labeled label of the sample pair, the sample pair comprising a first modality image and a second modality image; extracting modality features of the first modality image and the second modality image respectively, and obtaining key point detection results corresponding to the first modality image and the second modality image respectively according to the modality features corresponding to the first modality image and the second modality image respectively; obtaining a first difference between the key point detection results of the images of different modalities and the corresponding labeled labels, and obtaining a second difference between different modality features according to the modality features of the images of different modalities; optimizing the key point detection network in the direction of reducing the first difference and the second difference; and the trained key point detection network is used for key point detection of objects in images under any modality, which can improve the accuracy of the trained key point detection network.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Existing training methods for keypoint detection networks typically involve annotating acquired images and then iteratively training the network based on both the recognition and annotation results. Alternatively, multi-task training can be employed to improve the accuracy of keypoint recognition. However, these methods result in lengthy training cycles, and multi-task training requires different training methods for different tasks, making the training process cumbersome. Therefore, effectively training keypoint detection networks and improving their accuracy has become a current research hotspot. Summary of the Invention

[0003] This invention provides a model training method, apparatus, computer device, and storage medium, which can improve the accuracy of the trained keypoint detection network.

[0004] On one hand, embodiments of the present invention provide a model training method, including:

[0005] Obtain sample pairs and their annotation labels. Each sample pair includes a first modal image and a second modal image. The first modal image and the second modal image are cross-modal images obtained by acquiring images of objects in the same scene using different modalities. The annotation labels are used to mark the positions of key points of the object in the corresponding modal images.

[0006] Modal features are extracted from the first modal image and the second modal image respectively, and key point detection results corresponding to the first modal image and the second modal image are obtained based on the modal features corresponding to the first modal image and the second modal image respectively;

[0007] The first difference between the keypoint detection results and the corresponding labels of images of different modalities is obtained, and the second difference between the modal features of different modalities is obtained based on the modal features of images of different modalities.

[0008] The keypoint detection network is optimized in the direction of reducing the first difference and the second difference; the trained keypoint detection network is used to detect keypoints of objects in an image of any modality.

[0009] In another aspect, embodiments of the present invention provide a model training apparatus, comprising:

[0010] An acquisition unit is used to acquire sample pairs and their annotation labels. The sample pairs include a first modal image and a second modal image. The first modal image and the second modal image are cross-modal images obtained by acquiring images of objects in the same scene using different modalities. The annotation labels are used to mark the positions of key points of the object in the corresponding modal images.

[0011] The processing unit is configured to extract modal features from the first modal image and the second modal image respectively, and obtain key point detection results corresponding to the first modal image and the second modal image respectively based on the modal features corresponding to the first modal image and the second modal image respectively;

[0012] The acquisition unit is further configured to acquire the first difference between the key point detection results and the corresponding annotation labels of images of different modalities, and acquire the second difference between the modal features of images of different modalities based on the modal features of images of different modalities;

[0013] The processing unit is further configured to optimize the keypoint detection network in the direction of reducing the first difference and the second difference; the trained keypoint detection network is used to perform keypoint detection on objects in an image under any modality.

[0014] In another aspect, embodiments of the present invention provide a computer device, including a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program supporting the computer device in executing the above-described method, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the following steps:

[0015] Obtain sample pairs and their annotation labels. Each sample pair includes a first modal image and a second modal image. The first modal image and the second modal image are cross-modal images obtained by acquiring images of objects in the same scene using different modalities. The annotation labels are used to mark the positions of key points of the object in the corresponding modal images.

[0016] Modal features are extracted from the first modal image and the second modal image respectively, and key point detection results corresponding to the first modal image and the second modal image are obtained based on the modal features corresponding to the first modal image and the second modal image respectively;

[0017] The first difference between the keypoint detection results and the corresponding labels of images of different modalities is obtained, and the second difference between the modal features of different modalities is obtained based on the modal features of images of different modalities.

[0018] The keypoint detection network is optimized in the direction of reducing the first difference and the second difference; the trained keypoint detection network is used to detect keypoints of objects in an image of any modality.

[0019] In another aspect, embodiments of the present invention provide a computer-readable storage medium storing program instructions, which, when executed by a processor, are used to perform the model training method as described in the first aspect.

[0020] In this embodiment, after acquiring sample pairs composed of cross-modal images and their labels, the computer device can use an initially trained keypoint detection network to extract the modal features of the first and second modal images contained in the cross-modal images, as well as the keypoint detection results of each modal image. Then, based on the differences between the modal features corresponding to different modal images and the differences between the keypoint detection results and the keypoint annotation results in the labels, the computer device can further optimize and adjust the initially trained keypoint detection network to obtain a trained keypoint detection network. This enables the computer device to jointly optimize the keypoint detection network based on cross-modal images, thereby improving the accuracy of keypoint detection by the keypoint detection network. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1a This is a schematic diagram of an image of a different modality provided in an embodiment of the present invention;

[0023] Figure 1b This is a schematic diagram of a model training method provided in an embodiment of the present invention;

[0024] Figure 1c This is a schematic diagram of a key point detection method provided in an embodiment of the present invention;

[0025] Figure 2 This is a schematic flowchart of a model training method provided in an embodiment of the present invention;

[0026] Figure 3a This is a schematic diagram of a keypoint detection network that has undergone initial training, provided by an embodiment of the present invention;

[0027] Figure 3b This is a schematic diagram of a joint training method provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of a model training method provided in an embodiment of the present invention;

[0029] Figure 5 This is a schematic block diagram of a model training device provided in an embodiment of the present invention;

[0030] Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0031] This application proposes a model training method that allows a computer device to optimize and train a keypoint detection network using sample pairs of cross-modal images containing different modalities. In this method, one modality (e.g., a first modality image) provides higher accuracy for keypoint detection network optimization training, while another modality (e.g., a second modality image) provides lower accuracy. Due to the complementary nature of related data across different modalities, the computer device can improve the accuracy of the trained keypoint detection network for keypoint detection in images of the second modality after training the model using sample pairs of cross-modal images. In one embodiment, key point detection is used to detect and identify key points of the face or posture of a corresponding object (such as a person or animal). The results of key point detection can be used in the field of recognition and analysis, specifically in application scenarios such as expression analysis, posture judgment, quality analysis, 3D object reconstruction, and 3D animation. It should be noted that in the embodiments of this application, when collecting key points of a person's face or posture, the collection is only performed after informing the relevant person object and obtaining the corresponding collection authorization.

[0032] In one embodiment, the training process for the keypoint detection network is based on Artificial Intelligence (AI). AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Artificial intelligence technology is a comprehensive discipline involving a wide range of fields, including both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. Specifically, the training of the keypoint detection network mainly involves machine learning (ML). Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0033] In one embodiment, to improve the efficiency of keypoint detection (or keypoint acquisition) of relevant objects, a computer device can train a corresponding keypoint detection network to effectively acquire keypoints of relevant objects and improve keypoint acquisition efficiency. To improve the accuracy of the trained keypoint detection network in detecting keypoints in images of different modalities, the computer device can use sample pairs composed of cross-modal images to optimize the training of the keypoint detection network. Specifically, if the accuracy of the keypoint detection network in detecting keypoints in the first modal image included in the sample pair is higher than the accuracy in detecting keypoints in the second modal image included in the sample pair, then training the keypoint detection network using sample pairs composed of cross-modal images can improve the accuracy of the trained keypoint detection network in detecting keypoints in images of the second modality. In one embodiment, the first modal image may be a color image (i.e., an RGB image or a YUV image, etc.), and the second modal image may be an infrared image or an image in another modality where key points are not obvious. In this embodiment, the first modal image in the sample pair is an RGB image and the second modal image is an infrared image, which is described in detail. When the first modal image or the second modal image is an image acquired in another modality, the embodiments of this application can also be referred to. In one embodiment, the first modal image and the second modal image in the sample pair acquired by the computer device are both obtained by acquiring images of objects in the same scene using different modalities. That is to say, the first modal image and the second modal image are the same except for the difference in the image presentation modality. The first modal image may be as follows: Figure 1a The first modality is an RGB image marked with 10, while the second modality image can be as follows: Figure 1a The infrared image marked with 11.

[0034] The RGB image of this first mode can be obtained by capturing images of the current environment using appropriate high-definition image acquisition devices (such as cameras), such as... Figure 1a As shown in the RGB image marked with 10, the RGB image obtained by high-definition image acquisition equipment contains a large amount of data and the key points are obvious. Therefore, when computer equipment uses RGB images to train a key point detection network, the recognition accuracy of the trained key point detection network is relatively high. And as... Figure 1a The infrared image marked with 11 can be acquired using an infrared sensor. However, infrared images have characteristics such as poor exposure, unclear key points, and small data volume, resulting in a low accuracy rate for key point detection.

[0035] In one embodiment, when a computer device trains a keypoint detection network using sample pairs composed of cross-modal images, it may refer to, for example, Figure 1b The process is shown, and can be specifically executed in the following three stages:

[0036] Phase 1: Network initialization phase.

[0037] Because RGB images contain a large amount of data, computer devices can use RGB images to initialize and train a keypoint detection network, enabling the initially trained network to possess preliminary keypoint detection capabilities. This initial training of the keypoint detection network primarily involves coordinate regression training of keypoints. In one embodiment, the keypoint detection network is used to extract keypoints of objects contained in the input image and obtain the coordinate vectors of the detected keypoints. The coordinate vectors corresponding to the keypoints of objects in the input image obtained by the keypoint detection network can be obtained as follows: Figure 1c As shown, this keypoint detection network can specifically include Convolutional Neural Networks (CNNs), a convolution computation network, a ReLU nonlinear activation function computation network, and a pooling network. That is, by performing convolution, nonlinear activation, and pooling operations on the input image, the keypoint detection network can determine the locations of key points of objects in the corresponding input image.

[0038] Phase 2: Joint Training Phase.

[0039] The keypoint detection network obtained in Stage 1 is jointly optimized based on sample pairs (including RGB and infrared images). By introducing RGB image information to assist the training process of infrared images, the model's keypoint recognition capability for RGB images can be transferred to infrared images. In the joint training phase, the trained image scene classifier (or scene classifier) ​​can be used to constrain the feature maps (i.e., modal features corresponding to modal images) extracted by the keypoint detection network. This constrains the feature map distribution of infrared images to the same distribution as that of RGB images, enabling the keypoint detection network to effectively acquire information from RGB images and effectively transfer the network's recognition capability for RGB images to infrared images.

[0040] In one embodiment, the scene classifier is essentially a classification network used to classify the modal features extracted from images of different modalities. Specifically, the scene classifier may include a CNN network, convolutional computation networks, non-linear activation function computation networks, and pooling networks. That is, after performing corresponding calculations on the input image through the networks included in the scene classifier, it outputs the classification result of the modal features corresponding to the modal image. Furthermore, it should be noted that the network structure of the scene classifier can also be a fully connected network. This means that the scene classifier can adaptively adjust its network structure according to the input to obtain the output result for the corresponding input image. The classification result of the modal features corresponding to the modal image is used to indicate the modality of the image captured, such as an image captured in infrared mode or an image captured in RGB mode.

[0041] Phase 3: Fine-tuning phase (parameter adjustment phase).

[0042] The network parameters of the keypoint detection network obtained in stage 2 are fine-tuned based on RGB and infrared images. Specifically, the loss weight of the infrared image can be increased to make the trained keypoint detection network more compatible with the infrared scene, thereby effectively improving the ability of the trained keypoint detection network to identify key points in the modal images of the infrared scene.

[0043] The following describes the specific process of training a keypoint detection network using samples composed of cross-modal images on a computer device. Figure 2 As shown, the method may include:

[0044] S201, Obtain sample pairs and their labels. Each sample pair includes a first modality image and a second modality image.

[0045] S202, extract the modal features of the first modal image and the second modal image respectively, and obtain the key point detection results corresponding to the first modal image and the second modal image respectively based on the modal features corresponding to the first modal image and the second modal image respectively.

[0046] In steps S201 and S202, the first modal image and the second modal image are cross-modal images obtained by acquiring images of objects in the same scene using different modalities. The annotation label is used to mark the position of the object's key points in the corresponding modal image. In one embodiment, the sample pair acquired by the computer device can be any one of multiple sample pairs contained in the training sample set. The modal features corresponding to the modal image, as well as the key point detection results for the objects contained in the modal image, are obtained when the computer device calls the initially trained key point detection network to perform key point recognition processing on the corresponding modal image. The initially trained key point detection network is obtained by the computer device after initial training of the key point detection network (or key point detection unit module). After initial training, the network that does not have key point detection capability can learn a certain key point detection capability. In one embodiment, the process of obtaining the initially trained key point detection network can be as follows: Figure 3a As shown, the specific method can be as follows: The computer device can first select one or more first modality images from multiple sample pairs contained in the training sample set, and obtain the corresponding annotation label for each selected first modality image; wherein, the accuracy of keypoint detection network for keypoint detection of first modality images and second modality images is different, and the accuracy of keypoint detection network for keypoint detection of first modality images is higher than that for keypoint detection of second modality images. Then, after the computer device obtains one or more first modality images, it can use the initialized keypoint detection network to perform keypoint detection on each selected first modality image. Then, based on the keypoint detection results for each selected first modality image and the corresponding annotation label, the initialized keypoint detection network can be initially trained to obtain the initially trained keypoint detection network.

[0047] Below, in conjunction with Figure 3a The following describes the specific steps involved in obtaining the initial training of the keypoint detection network on the computer device, using the various functional modules involved:

[0048] ① The training data preparation module reads the first modality image in the training sample set and combines the read data into a training batch, which is then sent to the key point detection network for processing.

[0049] ② A keypoint detection network is used to extract keypoints of objects in the input image and output the coordinate vectors corresponding to the detected keypoints;

[0050] ③ In the regression task objective function calculation module, the module takes the keypoint coordinates detected by the keypoint detection network and the true coordinates of the keypoints indicated by the labels as input to calculate the objective function value. The objective function can be any type of distance metric (such as L1 normal form distance, L2 normal form distance, or Smoth_L1), or other types of objective functions.

[0051] In one embodiment, the true coordinates of keypoints are indicated by the annotation labels of sample pairs. Since each sample pair in the training sample set is obtained by simultaneously fixing different image sensors and capturing images of the same scene, after acquiring a sample pair from different image sensors, the computer device can annotate only one modality image (e.g., a first modality image or a second modality image) within that sample pair to obtain the annotation label corresponding to that modality image. The annotation label of the unannotated modality image can be directly used as the annotation label of the already annotated modality image. Since all images input to the keypoint detection network are first modality images, when determining the annotation label (i.e., keypoint annotation result) corresponding to each first modality image input to the keypoint detection network, if obtaining the annotation label corresponding to any selected first modality image fails, the computer device can determine the target sample pair corresponding to any first modality image. Then, it can obtain the second modality image associated with any first modality image from the target sample pair and use the annotation label obtained from the associated second modality image as the keypoint annotation result of the corresponding first modality image. By requiring only the annotation of one modality image, the manpower cost in the initial training phase can be reduced while ensuring the accuracy of the modality image annotation.

[0052] ④ The entire network is trained and optimized using gradient descent methods (such as stochastic gradient descent, stochastic gradient descent with a driving term, Adam (an optimization algorithm), and Adamard (another optimization algorithm)). In one embodiment, the keypoint detection result for any first modality image includes: the detected coordinate vectors of the object's keypoints in any first modality image (i.e., the coordinate vectors output by the keypoint detection network), and the annotation label for any first modality image includes: the labeled coordinate vectors of the object's true keypoints in any first modality image. Therefore, the process of the computer device performing initial training on the initialized keypoint detection network based on the keypoint detection results and corresponding annotation labels for each selected first modality image is equivalent to calculating the objective function value based on the detected coordinate vectors and labeled coordinate vectors, and then training the initialized keypoint detection network using the gradient descent algorithm based on the objective function value.

[0053] The computer equipment continuously trains the keypoint detection network by repeatedly applying steps ① to ④ with new samples from the training samples until the training result meets the training termination condition, thus obtaining the initially trained keypoint detection network. The condition for terminating model training (i.e., terminating network optimization training) is generally set by either meeting a predetermined number of iterations or ensuring that the loss value calculated by the objective function is less than a predetermined value.

[0054] In one embodiment, the initially trained keypoint detection network can also be obtained using model knowledge distillation technology. That is, the computer device can use a large-scale keypoint detection network with higher accuracy to perform knowledge distillation on the small-scale keypoint detection network, thereby improving the accuracy of the small-scale keypoint detection network. Based on the obtained initially trained keypoint detection network, the computer device can extract modal features from the first modal image and the second modal image in the sample pair, respectively, thereby obtaining the modal features of the first modal image and the modal features corresponding to the second modal image. In addition, the computer device can also obtain the keypoint detection results corresponding to the first modal image and the second modal image, respectively. In one embodiment, based on the modal features extracted from the modal images by the computer device and the keypoint detection results obtained by keypoint recognition, the keypoint detection network and the image scene classifier can be jointly trained, and then step S203 can be executed.

[0055] S203, obtain the first difference between the key point detection results of images of different modalities and the corresponding annotation labels, and obtain the second difference between the modal features of different modalities based on the modal features of images of different modalities.

[0056] In one embodiment, the computer device can perform joint training of a keypoint detection network and an image scene classifier as follows: Figure 3b As shown, the joint training process mainly consists of two training phases. The first phase involves training a better image scene classifier (or scene classifier). After training, the scene classifier can classify the feature maps (i.e., modal features) extracted by the keypoint detection network. The second phase involves introducing the scene classifier trained in the first phase into the joint optimization framework to further constrain the features of cross-modal images contained in the sample pairs.

[0057] In one embodiment, when training a scene classifier, the computer device combines the initially trained keypoint detection network with the scene classifier. However, during this training process, only the output of the keypoint detection network is needed; the network parameters of the keypoint detection network are not updated or optimized. The training process of the scene classifier by the computer device is as follows: the computer device first acquires a training sample set, which contains multiple sample pairs, each containing a first modality image and a second modality image. Then, it acquires the modal features of each modality image in the training sample set and calls the initialized scene classifier to perform recognition processing on the modal features of each modality image in the training sample set, obtaining a classification label for each modality image in the training sample set. In one embodiment, when acquiring modal features of each modal image, the computer device can input each modal image into an initially trained keypoint detection network, thereby obtaining the modal features corresponding to the modal image from the intermediate layer (or the last layer) of the initially trained keypoint detection network. After obtaining the modal features corresponding to each modal image in the training sample set, the computer device can obtain a classification label for the modal image based on the recognition processing of the modal features by the initialized scene classifier. This label can be used to indicate the modal type of the corresponding modal image, that is, the classification label can be used to indicate whether the corresponding modal image is an infrared image or an RGB image.

[0058] After the computer device obtains the classification labels of each modality image in the training sample set, the computer device can calculate the cross-entropy difference based on the classification labels and corresponding ground truth classification labels of each modality image in the training sample set. In one embodiment, the ground truth classification label can be stored in the annotation labels of the sample pairs, or it can be indicated by other labels. The computer device can calculate the cross-entropy difference through the cross-entropy objective function calculation module. The function of the cross-entropy objective function calculation module is to take the output of the scene classifier (i.e., the classification label) and the label of the modality to which the image belongs (i.e., the ground truth label) as input, and calculate the loss function value using the cross-entropy loss function calculation method. After the computer device calculates the cross-entropy difference (i.e., the cross-entropy loss value), the initial scene classifier can be trained using the gradient descent algorithm based on the cross-entropy difference. In a specific implementation, the computer device can train based on the cross-entropy difference through the cross-entropy objective function optimization module. In one embodiment, the cross-entropy objective function optimization module will train and optimize the scene classifier based on gradient descent (such as the stochastic gradient descent mentioned above, stochastic gradient descent with a driving term, Adam, Adamard, etc.), and can repeat the above steps using different samples in the training sample set until the training result of the scene classifier meets the training termination condition, thus obtaining the trained scene classifier.

[0059] After obtaining a trained scene classifier, the computer device can call the trained scene classifier to obtain a second difference between different modal features. In a specific implementation, the computer device can call the scene classifier to perform recognition processing on the modal features of the first modal image and the modal features of the second modal image, respectively, to obtain the classification labels of the first modal image and the second modal image. Then, based on the label difference between the classification labels of the first modal image and the second modal image, the second difference between the corresponding modal features can be determined. The value of the second difference is positively correlated with the feature difference between the corresponding modal features of the first modal image and the corresponding modal features of the second modal image.

[0060] In one embodiment, when jointly training an initially trained keypoint detection network and an image scene classifier, after inputting each modality image from the training sample set into the initially trained keypoint detection network, the network will output the keypoint detection results (i.e., the position vectors of keypoints of objects in each modality image) for each modality image. This allows the computer device to further combine the corresponding annotation labels of each modality image to determine the first difference between the keypoint detection results and the corresponding annotation labels of different modalities. This first difference can be obtained by a training data preparation module, whose function is the same as that used during the initial training of the keypoint detection network. This training data preparation module reads data from different modalities corresponding to the same image (i.e., a sample pair), and the target five-point labels (i.e., the corresponding annotation labels) in the different modalities of the sample pair are the same.

[0061] Since a sample pair contains images of at least two modalities, the first difference obtained by the computer device includes the first difference between the first modal image and the labeled image, obtained based on the keypoint detection results and labels corresponding to the first modal image; and the first difference between the second modal image and the labeled image, obtained based on the keypoint detection results and labels corresponding to the second modal image. After the computer device determines the first difference and the second difference, it can optimize the keypoint detection network in the direction of reducing the first difference and the second difference, i.e., proceed to step S204. In this embodiment, the first difference or the second difference can refer to the loss value of the loss function constructed from the corresponding difference.

[0062] S204, Optimize the keypoint detection network in the direction of reducing the first and second differences; The trained keypoint detection network is used to detect keypoints of objects in an image of any modality.

[0063] When a computer device optimizes a keypoint detection network in the direction of reducing the first difference and the second difference, it can construct a target difference based on the first difference between the first modality image and the label, the first difference between the second modality image and the label, and the difference of the second difference. Then, the keypoint detection network can be optimized by reducing the target difference. In one embodiment, if the first modal image is an RGB image and the second modal image is an infrared image, then the first difference between the first modal image and the label is used to indicate the keypoint detection network's ability to detect keypoints in the RGB image, while the first difference between the second modal image and the label is used to indicate the keypoint detection network's ability to detect keypoints in the infrared image. It can be understood that the smaller the first difference between the first modal image and the label, the stronger the keypoint detection network's ability to detect keypoints in the RGB image, and the smaller the first difference between the second modal image and the label, the stronger the keypoint detection network's ability to detect keypoints in the infrared image. The second difference can be used to indicate the degree of attention the keypoint detection network pays to modal features when performing keypoint detection. Therefore, the computer device optimizes the keypoint detection network by combining the target difference formed by the first difference between the first modal image and the label, the first difference between the second modal image and the label, and the second difference. This is a process of coordinating the keypoint detection network to improve keypoint recognition performance and the degree of attention paid to modal features during keypoint recognition.

[0064] In one embodiment, the first difference between the first modality image and the labeled image can be represented by a first loss value (L1), which can be the difference calculated based on the coordinate vector of the detected keypoint and the true coordinate vector of the keypoint. The first difference between the second modality image and the labeled image can be represented by a second loss value (L2), and the second difference can be represented by a third loss value (L3). Then, the target difference determined by the computer device for optimizing the training of the keypoint detection network can be expressed as L = L1 + L2 + L3, so that the computer device can obtain the trained keypoint detection network by continuously reducing the value of the target difference L.

[0065] In one embodiment, the process of optimizing the keypoint detection network by reducing the first and second differences is also a process of jointly training the keypoint detection model using cross-modal images. When the computer device jointly trains the keypoint detection model using cross-modal images, a gradient backpropagation training layer can be introduced for joint training. During the joint training process using the gradient backpropagation training layer, the computer device can call a scene classifier to classify the features generated by the keypoint detection network (i.e., the modal features corresponding to different modal images). The classification result is superimposed on the backbone model (i.e., the keypoint detection network) in a reverse manner, so that the features extracted by the backbone model do not contain information that can be used to distinguish scenes, so as to learn the common information between different scenes. This allows the computer device to improve the infrared scene using the better performance of RGB. In this embodiment, the gradient backpropagation training layer has the same input and output when it performs forward pass. When it performs gradient update, the gradient after passing through this layer is adjusted. The mathematical expression formula of the gradient backpropagation training layer is shown in Equation 1 in one embodiment.

[0066]

[0067] Where X is the input feature of the gradient inverse training layer, R(X) is the output of the gradient inverse training layer, γ is a function that changes with the number of iterations p, I is the gradient, and when k is the total number of iterations, the value of γ is as shown in Equation 2:

[0068]

[0069] In one embodiment, based on the introduction of a gradient inverse training layer, the computer device optimizes the keypoint detection network in the direction of reducing the first and second differences. This involves first obtaining the target gradient adjustment function corresponding to the gradient inverse training layer (including the gradient functions indicated by Equations 1 and 2 above), and then further reducing the first and second differences according to the adjustment direction indicated by the target gradient adjustment function, thereby optimizing the keypoint detection network. Since the sample pairs trained by the computer device during the joint training phase are any one of the multiple sample pairs contained in the training sample set, when training the keypoint detection network based on each sample pair in the training sample set, the computer device needs to first calculate the first and second differences for each sample pair in the training sample set. Then, it can adjust the network parameters of the keypoint detection network in the direction of reducing the first and second differences corresponding to each sample pair. When the difference between the modal features corresponding to different modal images identified by the adjusted keypoint detection network is less than a threshold, the optimization of the keypoint detection network is considered complete. The computer device can determine that the difference between modal features corresponding to different modal images is less than a threshold when the modal features corresponding to different modal images all satisfy the same feature distribution. That is, the computer device can constrain the modal features corresponding to different modal images to the same feature distribution, so that the key point detection network only focuses on the key points of the objects contained in the image during optimization training, and does not focus on the modal features corresponding to the image. This can effectively improve the accuracy of the trained key point detection network in key point recognition. In addition, since the scene classifier recognizes different features of infrared scenes and RGB scenes, and uses the inverse gradient optimization method in the model backbone to constrain the key point model, the model obtains a feature expression consistent with the RGB scene distribution, which can effectively improve the accuracy of the trained key point detection network in key point detection in infrared scenes.

[0070] To further improve the accuracy of keypoint detection networks, the computer device can fine-tune the network parameters of the keypoint detection network after jointly training the scene classifier and the keypoint detection network using samples from the training sample set. Specifically, the computer device can further adjust the network parameters of the keypoint detection network by adjusting the target difference constructed from the first and second differences. As mentioned above, the target difference constructed by the computer device based on the first and second differences can be expressed as the sum of the differences between the first modality image and the labeled image, the first difference between the second modality image and the labeled image, and the second difference, i.e., the target difference can be expressed as L = L1 + L2 + L3. When adjusting this target difference, the computer device can fine-tune the model using RGB images (i.e., the first modality image) and infrared images (i.e., the second modality image), increasing the loss weight of the infrared image. This makes the keypoint detection network more closely match the infrared scene, thus improving the accuracy of keypoint recognition of objects in the infrared modality using the trained keypoint detection network.

[0071] When determining the target difference during the fine-tuning phase, the computer device can obtain the first weight corresponding to the first difference between the first modal image and the label, the second weight corresponding to the first difference between the second modal image and the label, and the third weight corresponding to the second difference. The absolute value of the second weight is greater than the absolute value of the first weight, and the absolute value of the second weight is greater than the absolute value of the third weight. Then, the computer device can weight the first difference between the first modal image and the label according to the first weight, weight the first difference between the second modal image and the label according to the second weight, and weight the second difference according to the third weight. The weighted differences are then summed to obtain the target difference. In other words, when fine-tuning the network parameters of the keypoint detection network in this stage, the target difference (i.e., loss function) used is L = αL1 + βL2 + γL3. Here, the second weight α corresponding to the supervision function (i.e., the difference) obtained from the infrared image (i.e., the second modality image) is increased, with an empirical value of approximately 1.5; the first weight β corresponding to the supervision function (i.e., the difference) obtained from the RGB image (i.e., the first modality image) is decreased, with an empirical value of 0.8; and the third weight γ corresponding to the supervision function obtained from the scene classifier is decreased, with an empirical value of 0.5. In one embodiment, when fine-tuning the network parameters of the keypoint detection network, the computer device reduces the optimized learning rate parameter and only fine-tunes the network. After the network parameters of the keypoint detection network are fine-tuned, it can be determined that the keypoint detection network has been trained. Then, the trained keypoint detection network can be deployed to various devices that need to perform keypoint recognition, so that keypoint recognition processing can be performed through the trained keypoint detection network.

[0072] In one embodiment, when training a keypoint detection network, the computer device can also directly follow the fine-tuning stage in the process of constructing target differences based on the first and second differences, by first determining the weights corresponding to each difference and constructing target differences based on each difference and its corresponding weights. Then, after obtaining the target differences constructed based on the weights, the computer device can directly train the keypoint detection network according to the constructed target differences, thereby obtaining the trained keypoint detection network.

[0073] Below, in conjunction with Figure 4 This paper provides a comprehensive explanation of the various stages of training a keypoint detection network using computer equipment, as well as the modules and units involved:

[0074] First, the training data preparation module can read the first modal image from multiple sample pairs contained in the training sample set, and use the first modal image to perform initial training on the key point detection network to obtain the initially trained key point detection network. Then, the initially trained key point detection network can be called to identify the modal features of the modal images contained in each sample pair in the training sample set to obtain the modal features corresponding to each modal image. Furthermore, the scene classification network can perform modal feature recognition processing to determine the second difference between the modal features corresponding to each modal image in the sample pair. The value of this second difference is positively correlated with the difference between the modal features corresponding to different modal images. That is, the greater the difference between the modal features corresponding to different modal images, the greater the determined second difference. Conversely, the smaller the difference between the modal features corresponding to different modal images, the smaller the determined second difference. By constructing the second difference and continuously reducing it through training, the computer device can make the differences between the modal features recognized by the keypoint detection network smaller and smaller. This allows the keypoint detection network to focus mainly on keypoint features during keypoint detection, without including information used for modal scene differentiation, thereby improving the recognition accuracy of the keypoint detection network.

[0075] Furthermore, the initially trained keypoint detection network will also perform keypoint detection and extraction on each sample pair in the training sample set. Based on the keypoint detection results and the corresponding annotation results, a first difference is determined. It can be understood that a sample pair contains images from at least two modalities; therefore, this first difference includes the detection differences corresponding to images from different modalities. Figure 4The image shows the detection difference L1 corresponding to the RGB image and the detection difference L2 corresponding to the infrared image. After the computer device acquires the corresponding differences, a target difference can be constructed based on the acquired differences. The keypoint detection network is then optimized and trained using gradient backpropagation and the target difference. The above steps are iterated continuously using each sample pair in the training sample set to obtain the trained keypoint recognition network.

[0076] In this embodiment, after acquiring sample pairs composed of cross-modal images and their labels, the computer device can use an initially trained keypoint detection network to extract the modal features of the first and second modal images contained in the cross-modal images, as well as the keypoint detection results of each modal image. Then, based on the differences between the modal features corresponding to different modal images and the differences between the keypoint detection results and the keypoint annotation results in the labels, the computer device can further optimize and adjust the initially trained keypoint detection network to obtain a trained keypoint detection network. This enables the computer device to jointly optimize the keypoint detection network based on cross-modal images, thereby improving the accuracy of keypoint detection by the keypoint detection network.

[0077] Based on the description of the above-described model training method embodiments, this invention also proposes a model training apparatus, which can be a computer program (including program code) running on the aforementioned computer device. This model training apparatus can be used to execute, for example... Figure 2 For the model training method described above, please refer to [link / reference]. Figure 5 The model training device includes an acquisition unit 501 and a processing unit 502.

[0078] The acquisition unit 501 is used to acquire sample pairs and their annotation labels. The sample pairs include a first modal image and a second modal image. The first modal image and the second modal image are cross-modal images obtained by acquiring images of objects in the same scene using different modalities. The annotation labels are used to mark the positions of key points of the object in the corresponding modal images.

[0079] The processing unit 502 is used to extract the modal features of the first modal image and the second modal image respectively, and obtain the key point detection results corresponding to the first modal image and the second modal image respectively based on the modal features corresponding to the first modal image and the second modal image respectively;

[0080] The acquisition unit 501 is further configured to acquire the first difference between the key point detection results and the corresponding annotation labels of images of different modalities, and acquire the second difference between the modal features of different modalities based on the modal features of images of different modalities.

[0081] The processing unit 502 is further configured to optimize the key point detection network in the direction of reducing the first difference and the second difference; the trained key point detection network is used to perform key point detection on objects in an image under any modality.

[0082] In one embodiment, the sample pair is any one of multiple sample pairs included in the training sample set, and both the modal features and the keypoint detection results are obtained by calling the initially trained keypoint detection network. The accuracy of keypoint detection for the first modal image and the second modal image differs depending on whether the keypoint detection network is used. If the accuracy of keypoint detection for the first modal image is higher than the accuracy of keypoint detection for the second modal image, the processing unit 502 is further specifically used for:

[0083] One or more first modality images are selected from multiple sample pairs contained in the training sample set, and the annotation label corresponding to each selected first modality image is obtained;

[0084] An initialized keypoint detection network is used to perform keypoint detection on each selected first modality image;

[0085] Based on the keypoint detection results and corresponding labels for each selected first modality image, the initialized keypoint detection network is initially trained to obtain the initially trained keypoint detection network.

[0086] In one embodiment, the processing unit 502 is further configured to determine the target sample pair corresponding to the selected first modality image if it fails to obtain the annotation label corresponding to any of the selected first modality images.

[0087] The processing unit 502 is further configured to obtain a second modal image associated with any of the first modal images from the target sample pair, and use the annotation labels obtained from the associated second modal images as the key point annotation results of any of the first modal images.

[0088] In one embodiment, the keypoint detection result for any first modality image includes: the detected coordinate vectors of the object's keypoints in the first modality image, and the annotation label of any first modality image includes: the labeled coordinate vectors of the object's true keypoints in the first modality image; the processing unit 502 is specifically used for:

[0089] The objective function value is calculated based on the detected coordinate vector and the labeled coordinate vector, and the objective function value includes the distance value calculated based on each detected coordinate vector and the corresponding labeled coordinate vector.

[0090] Based on the objective function value, the initial keypoint detection network is trained using the gradient descent algorithm.

[0091] In one embodiment, the acquisition unit 501 is specifically used for:

[0092] The scene classifier is invoked to recognize the modal features of the first modal image and the modal features of the second modal image respectively, so as to obtain the classification label of the first modal image and the classification label of the second modal image;

[0093] Based on the label difference between the classification labels of the first modality image and the classification labels of the second modality image, a second difference between corresponding modal features is determined.

[0094] In one embodiment, the scene classifier is obtained after training; the acquisition unit 501 is also used to acquire a training sample set, which contains multiple sample pairs, each sample pair containing a first modality image and a second modality image.

[0095] The acquisition unit 501 is further configured to acquire the modal features of each modal image contained in the training sample set, and call the initialized scene classifier to perform recognition processing on the modal features of each modal image contained in the training sample set, so as to obtain the classification label of each modal image in the training sample set.

[0096] The cross-entropy difference is calculated based on the classification label and the corresponding true classification label of each modality image in the training sample set, and the initial scene classifier is trained using the gradient descent algorithm based on the cross-entropy difference.

[0097] In one embodiment, the acquisition unit 501 is specifically used for:

[0098] Based on the key point detection results corresponding to the first modality image and the annotation label, a first difference between the first modality image and the annotation label is obtained; and,

[0099] Based on the key point detection results corresponding to the second modality image and the annotation label, the first difference between the second modality image and the annotation label is obtained.

[0100] In one embodiment, the processing unit 502 is specifically used for:

[0101] Based on the first difference between the first modal image and the annotation label, the first difference between the second modal image and the annotation label, and the difference between the second difference and the target difference;

[0102] The keypoint detection network is optimized by reducing the target differences.

[0103] In one embodiment, if the accuracy of the keypoint detection network in detecting keypoints in the first modality image is higher than the accuracy in detecting keypoints in the second modality image, the processing unit 502 is specifically used for:

[0104] A first weight corresponding to the first difference between the first modal image and the label, a second weight corresponding to the first difference between the second modal image and the label, and a third weight corresponding to the second difference are obtained respectively; wherein, the absolute value of the second weight is greater than the absolute value of the first weight, and the absolute value of the second weight is greater than the absolute value of the third weight;

[0105] The first difference between the first modal image and the label is weighted according to the first weight, the first difference between the second modal image and the label is weighted according to the second weight, and the second difference is weighted according to the third weight. The weighted differences are summed to obtain the target difference.

[0106] In one embodiment, the processing unit 502 is specifically used for:

[0107] Obtain the target gradient adjustment function corresponding to the gradient inverse training layer;

[0108] The first difference and the second difference are reduced according to the adjustment direction indicated by the target gradient adjustment function in order to optimize the key point detection network.

[0109] In one embodiment, the sample pair is any one of multiple sample pairs included in the training sample set; the processing unit 502 is specifically used for:

[0110] For each pair of samples in the training sample set, calculate the first difference and the second difference for each pair of samples;

[0111] By reducing the direction of the first and second differences corresponding to each sample pair, the network parameters of the key point detection network are adjusted, and the optimization of the key point detection network is determined to be complete when the difference between the modal features corresponding to the different modal images identified by the adjusted key point detection network is less than a threshold.

[0112] In one embodiment, the processing unit 502 is further configured to: determine that the difference between the modal features corresponding to the different modal images is less than a threshold when the modal features corresponding to the different modal images all satisfy the same feature distribution.

[0113] In this embodiment, after the acquisition unit 501 acquires the sample pairs composed of cross-modal images and their labels, the processing unit 502 can use the initially trained keypoint detection network to extract the modal features of the first and second modal images contained in the cross-modal images, as well as the keypoint detection results of each modal image. Then, the processing unit 502 can further optimize and adjust the initially trained keypoint detection network based on the differences between the modal features corresponding to different modal images and the differences between the keypoint detection results and the keypoint annotation results in the labels, thereby obtaining the trained keypoint detection network. This achieves joint optimization of the keypoint detection network based on cross-modal images, which can improve the accuracy of keypoint detection by the keypoint detection network.

[0114] Please see Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device can be a server or a terminal device, wherein the terminal device includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. Figure 6 The computer device shown in this embodiment may include: one or more processors 601; one or more input devices 602; one or more output devices 603; and a memory 604. The processors 601, input devices 602, output devices 603, and memory 604 are connected via a bus 605. The memory 604 stores a computer program, which includes program instructions, and the processor 601 executes the program instructions stored in the memory 604.

[0115] The memory 604 may include volatile memory, such as random-access memory (RAM); the memory 604 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 604 may also include a combination of the above types of memory.

[0116] The processor 601 may be a central processing unit (CPU). The processor 601 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc. The PLD may be a field-programmable gate array (FPGA), generic array logic (GAL), etc. The processor 601 may also be a combination of the above structures.

[0117] In this embodiment of the invention, the memory 604 is used to store a computer program, the computer program including program instructions, and the processor 601 is used to execute the program instructions stored in the memory 604 to implement the above-mentioned... Figure 2 The steps of the corresponding method.

[0118] In one embodiment, the processor 601 is configured to invoke the program instructions to execute:

[0119] Obtain sample pairs and their annotation labels. Each sample pair includes a first modal image and a second modal image. The first modal image and the second modal image are cross-modal images obtained by acquiring images of objects in the same scene using different modalities. The annotation labels are used to mark the positions of key points of the object in the corresponding modal images.

[0120] Modal features are extracted from the first modal image and the second modal image respectively, and key point detection results corresponding to the first modal image and the second modal image are obtained based on the modal features corresponding to the first modal image and the second modal image respectively;

[0121] The first difference between the keypoint detection results and the corresponding labels of images of different modalities is obtained, and the second difference between the modal features of different modalities is obtained based on the modal features of images of different modalities.

[0122] The keypoint detection network is optimized in the direction of reducing the first difference and the second difference; the trained keypoint detection network is used to detect keypoints of objects in an image of any modality.

[0123] In one embodiment, the sample pair is any one of multiple sample pairs included in the training sample set. Both the modal features and the keypoint detection results are obtained by calling the initially trained keypoint detection network. The accuracy of keypoint detection for the first modal image and the second modal image differs depending on whether the keypoint detection network is used. If the accuracy of keypoint detection for the first modal image is higher than the accuracy of keypoint detection for the second modal image, the processor 601 is configured to call the program instructions to execute:

[0124] One or more first modality images are selected from multiple sample pairs contained in the training sample set, and the annotation label corresponding to each selected first modality image is obtained;

[0125] An initialized keypoint detection network is used to perform keypoint detection on each selected first modality image;

[0126] Based on the keypoint detection results and corresponding labels for each selected first modality image, the initialized keypoint detection network is initially trained to obtain the initially trained keypoint detection network.

[0127] In one embodiment, the processor 601 is configured to invoke the program instructions to execute:

[0128] If it fails to obtain the label corresponding to any of the selected first modal images, then the target sample pair corresponding to the first modal image is determined.

[0129] Obtain a second modal image associated with any of the first modal images from the target sample pairs, and use the annotation labels obtained from the associated second modal images as the key point annotation results of any of the first modal images.

[0130] In one embodiment, the keypoint detection result for any first modality image includes: the detected coordinate vectors of the object's keypoints in the first modality image, and the annotation label of any first modality image includes: the labeled coordinate vectors of the object's true keypoints in the first modality image; the processor 601 is configured to call the program instructions to execute:

[0131] The objective function value is calculated based on the detected coordinate vector and the labeled coordinate vector, and the objective function value includes the distance value calculated based on each detected coordinate vector and the corresponding labeled coordinate vector.

[0132] Based on the objective function value, the initial keypoint detection network is trained using the gradient descent algorithm.

[0133] In one embodiment, the processor 601 is configured to invoke the program instructions to execute:

[0134] The scene classifier is invoked to recognize the modal features of the first modal image and the modal features of the second modal image respectively, so as to obtain the classification label of the first modal image and the classification label of the second modal image;

[0135] Based on the label difference between the classification labels of the first modality image and the classification labels of the second modality image, a second difference between corresponding modal features is determined.

[0136] In one embodiment, the scene classifier is obtained after training; the processor 601 is configured to invoke the program instructions to execute:

[0137] Obtain a training sample set, which contains multiple sample pairs, each of which contains a first modality image and a second modality image;

[0138] The modal features of each modal image contained in the training sample set are obtained, and the initialized scene classifier is called to perform recognition processing on the modal features of each modal image contained in the training sample set to obtain the classification label of each modal image in the training sample set.

[0139] The cross-entropy difference is calculated based on the classification label and the corresponding true classification label of each modality image in the training sample set, and the initial scene classifier is trained using the gradient descent algorithm based on the cross-entropy difference.

[0140] In one embodiment, the processor 601 is configured to invoke the program instructions to execute:

[0141] Based on the key point detection results corresponding to the first modality image and the annotation label, a first difference between the first modality image and the annotation label is obtained; and,

[0142] Based on the key point detection results corresponding to the second modality image and the annotation label, the first difference between the second modality image and the annotation label is obtained.

[0143] In one embodiment, the processor 601 is configured to invoke the program instructions to execute:

[0144] Based on the first difference between the first modal image and the annotation label, the first difference between the second modal image and the annotation label, and the difference between the second difference and the target difference;

[0145] The keypoint detection network is optimized by reducing the target differences.

[0146] In one embodiment, if the accuracy of the keypoint detection network in detecting keypoints on the first modality image is higher than the accuracy in detecting keypoints on the second modality image, the processor 601 is configured to invoke the program instructions to execute:

[0147] A first weight corresponding to the first difference between the first modal image and the label, a second weight corresponding to the first difference between the second modal image and the label, and a third weight corresponding to the second difference are obtained respectively; wherein, the absolute value of the second weight is greater than the absolute value of the first weight, and the absolute value of the second weight is greater than the absolute value of the third weight;

[0148] The first difference between the first modal image and the label is weighted according to the first weight, the first difference between the second modal image and the label is weighted according to the second weight, and the second difference is weighted according to the third weight. The weighted differences are summed to obtain the target difference.

[0149] In one embodiment, the processor 601 is configured to invoke the program instructions to execute:

[0150] Obtain the target gradient adjustment function corresponding to the gradient inverse training layer;

[0151] The first difference and the second difference are reduced according to the adjustment direction indicated by the target gradient adjustment function in order to optimize the key point detection network.

[0152] In one embodiment, the sample pair is any one of multiple sample pairs included in the training sample set; the processor 601 is configured to invoke the program instructions to execute:

[0153] For each pair of samples in the training sample set, calculate the first difference and the second difference for each pair of samples;

[0154] By reducing the direction of the first and second differences corresponding to each sample pair, the network parameters of the key point detection network are adjusted, and the optimization of the key point detection network is determined to be complete when the difference between the modal features corresponding to the different modal images identified by the adjusted key point detection network is less than a threshold.

[0155] When the modal features corresponding to the different modal images all satisfy the same feature distribution, it is determined that the difference between the modal features corresponding to the different modal images is less than a threshold.

[0156] This invention provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned actions. Figure 2 The method embodiment shown. The computer-readable storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0157] The above-disclosed embodiments are merely partial examples of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments, and equivalent variations made in accordance with the claims of the present invention, still fall within the scope of the invention.

Claims

1. A model training method, characterized in that, include: Obtain sample pairs and their labels, wherein the sample pairs include a first modality image and a second modality image; The first modal image and the second modal image are cross-modal images obtained by acquiring images of objects in the same scene using different modalities. The annotation labels are used to mark the positions of the key points of the object in the corresponding modal images. Modal features are extracted from the first modal image and the second modal image respectively, and key point detection results corresponding to the first modal image and the second modal image are obtained based on the modal features corresponding to the first modal image and the second modal image respectively; The first difference between the keypoint detection results and the corresponding labels of images of different modalities is obtained, and the second difference between the modal features of different modalities is obtained based on the modal features of images of different modalities. The keypoint detection network is optimized in the direction of reducing the first difference and the second difference; The trained keypoint detection network is used to detect keypoints in objects in images of any modality. The sample pair is any one of multiple sample pairs contained in the training sample set. Both the modal features and the keypoint detection results are obtained by calling the initially trained keypoint detection network. The accuracy of keypoint detection for the first modal image and the second modal image differs depending on the keypoint detection network used. If the accuracy of keypoint detection for the first modal image is higher than that for the second modal image, the method for obtaining the initially trained keypoint detection network includes: selecting one or more first modal images from the multiple sample pairs contained in the training sample set and obtaining the corresponding label for each selected first modal image; using the initialized keypoint detection network to perform keypoint detection on each selected first modal image; and performing initial training on the initialized keypoint detection network based on the keypoint detection results and corresponding labels for each selected first modal image to obtain the initially trained keypoint detection network.

2. The method as described in claim 1, characterized in that, The method further includes: If it fails to obtain the label corresponding to any of the selected first modal images, then the target sample pair corresponding to the first modal image is determined. Obtain a second modal image associated with any of the first modal images from the target sample pairs, and use the annotation labels obtained from the associated second modal images as the key point annotation results of any of the first modal images.

3. The method as described in claim 1, characterized in that, The key point detection result for any first modality image includes: the detected coordinate vector of the key points of the object in any first modality image; the annotation label of any first modality image includes: the annotation coordinate vector of the true key points of the object in any first modality image. The initial training of the initialized keypoint detection network based on the keypoint detection results and corresponding labels for each selected first modality image includes: The objective function value is calculated based on the detected coordinate vector and the labeled coordinate vector, and the objective function value includes the distance value calculated based on each detected coordinate vector and the corresponding labeled coordinate vector. Based on the objective function value, the initial keypoint detection network is trained using the gradient descent algorithm.

4. The method as described in claim 1, characterized in that, The acquisition of the second difference between different modal features includes: The scene classifier is invoked to recognize the modal features of the first modal image and the modal features of the second modal image respectively, so as to obtain the classification label of the first modal image and the classification label of the second modal image; Based on the label difference between the classification labels of the first modality image and the classification labels of the second modality image, a second difference between corresponding modal features is determined.

5. The method as described in claim 4, characterized in that, The scene classifier is obtained after training; the method further includes: Obtain a training sample set, which contains multiple sample pairs, each of which contains a first modality image and a second modality image; The modal features of each modal image contained in the training sample set are obtained, and the initialized scene classifier is called to perform recognition processing on the modal features of each modal image contained in the training sample set to obtain the classification label of each modal image in the training sample set. The cross-entropy difference is calculated based on the classification label and the corresponding true classification label of each modality image in the training sample set, and the initial scene classifier is trained using the gradient descent algorithm based on the cross-entropy difference.

6. The method as described in claim 1, characterized in that, Obtain the first difference between the keypoint detection results and the corresponding annotation labels of images of different modalities, including: Based on the key point detection results corresponding to the first modality image and the annotation label, a first difference between the first modality image and the annotation label is obtained; and, Based on the key point detection results corresponding to the second modality image and the annotation label, the first difference between the second modality image and the annotation label is obtained.

7. The method as described in claim 6, characterized in that, The optimization of the keypoint detection network in the direction of reducing the first difference and the second difference includes: Based on the first difference between the first modal image and the annotation label, the first difference between the second modal image and the annotation label, and the difference between the second difference and the target difference; The keypoint detection network is optimized by reducing the target differences.

8. The method as described in claim 7, characterized in that, If the accuracy of the keypoint detection network in detecting keypoints on the first modality image is higher than the accuracy in detecting keypoints on the second modality image; the step of constructing a target difference based on the first difference between the first modality image and the annotation label, the first difference between the second modality image and the annotation label, and the difference of the second difference and the target difference includes: A first weight corresponding to the first difference between the first modal image and the label, a second weight corresponding to the first difference between the second modal image and the label, and a third weight corresponding to the second difference are obtained respectively; wherein, the absolute value of the second weight is greater than the absolute value of the first weight, and the absolute value of the second weight is greater than the absolute value of the third weight; The first difference between the first modal image and the label is weighted according to the first weight, the first difference between the second modal image and the label is weighted according to the second weight, and the second difference is weighted according to the third weight. The weighted differences are summed to obtain the target difference.

9. The method as described in claim 1, characterized in that, The optimization of the keypoint detection network in the direction of reducing the first difference and the second difference includes: Obtain the target gradient adjustment function corresponding to the gradient inverse training layer; The first difference and the second difference are reduced according to the adjustment direction indicated by the target gradient adjustment function in order to optimize the key point detection network.

10. The method as described in claim 1, characterized in that, The sample pair is any one of multiple sample pairs contained in the training sample set; the optimization of the keypoint detection network in the direction of reducing the first difference and the second difference includes: For each pair of samples in the training sample set, calculate the first difference and the second difference for each pair of samples; By reducing the direction of the first and second differences corresponding to each sample pair, the network parameters of the key point detection network are adjusted, and the optimization of the key point detection network is determined to be complete when the difference between the modal features corresponding to the different modal images identified by the adjusted key point detection network is less than a threshold.

11. The method as described in claim 10, characterized in that, The method further includes: When the modal features corresponding to the different modal images all satisfy the same feature distribution, it is determined that the difference between the modal features corresponding to the different modal images is less than a threshold.

12. A model training device, characterized in that, include: An acquisition unit is used to acquire sample pairs and their labels, wherein the sample pairs include a first modality image and a second modality image; The first modal image and the second modal image are cross-modal images obtained by acquiring images of objects in the same scene using different modalities. The annotation labels are used to mark the positions of the key points of the object in the corresponding modal images. The processing unit is configured to extract modal features from the first modal image and the second modal image respectively, and obtain key point detection results corresponding to the first modal image and the second modal image respectively based on the modal features corresponding to the first modal image and the second modal image respectively; The acquisition unit is further configured to acquire the first difference between the key point detection results and the corresponding annotation labels of images of different modalities, and acquire the second difference between the modal features of images of different modalities based on the modal features of images of different modalities; The processing unit is further configured to optimize the key point detection network in the direction of reducing the first difference and the second difference; The trained keypoint detection network is used to detect keypoints in objects in an image of any modality. The sample pair is any one of multiple sample pairs contained in the training sample set. The modality features and the keypoint detection results are both obtained by calling the initially trained keypoint detection network. The accuracy of keypoint detection for the first modality image and the second modality image is different. If the accuracy of keypoint detection for the first modality image is higher than that for the second modality image, the method of obtaining the initially trained keypoint detection network includes: selecting one or more first modality images from the multiple sample pairs contained in the training sample set and obtaining the label corresponding to each selected first modality image; using the initialized keypoint detection network to perform keypoint detection on each selected first modality image; and performing initial training on the initialized keypoint detection network based on the keypoint detection results and corresponding labels for each selected first modality image to obtain the initially trained keypoint detection network.

13. A computer device, characterized in that, The device includes a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 11.

Images