Key point understanding method, model training method and related device

By obtaining image and position prompt information, and using large language models for encoding and decoding, the semantic understanding and positioning of key points are realized, the problem of insufficient accuracy and efficiency of key point detection in computer vision is solved, and the key points in the image is improved The accuracy of detection and the efficiency of model training.

CN119027689APending Publication Date: 2024-11-26SHENZHEN TETRAS AI TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410868606.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the field of computer vision, it is difficult for the prior art to effectively realize the semantic understanding and positioning of key points in images, resulting in insufficient accuracy and efficiency of key point detection.

Method used

By obtaining the position prompt information of the image to be detected and the target key points, using a large language model for encoding and decoding, semantic information is obtained to locate key points, and image processing is carried out in combination with semantic understanding and position prompt information to realize the semantic understanding and position of key points. Test.

Benefits of technology

The accuracy and efficiency of key point detection in the image are improved, and the accuracy of key point detection and model training are improved through the preliminary identification and position judgment of semantic information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027689A_ABST
    Figure CN119027689A_ABST
Patent Text Reader

Abstract

The invention discloses a key point understanding method, a model training method and a related device, and the method comprises the steps: obtaining a to-be-detected image, and obtaining the position prompt information of a target key point; semantic understanding is conducted on the to-be-detected image based on the position prompt information, semantic information of the target key point in the to-be-detected image is obtained, the semantic information is used for determining the position information of the target key point in the to-be-detected image, and semantic understanding of the key point in the image can be achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer vision processing, and in particular to a key point understanding method, a model training method and related devices. Background Art

[0002] In recent years, with the continuous development of artificial intelligence technology and its application in all aspects of production and life, computer vision technology, as an important branch of artificial intelligence technology, has also received widespread attention. Computer vision technology involves the computer's understanding and analysis of images.

[0003] The key points contained in the image are the focus of attention in the field of computer vision. Therefore, how to achieve key point understanding has become a problem that needs to be solved urgently. Summary of the invention

[0004] This application at least provides a key point understanding method, a model training method and related devices.

[0005] The present application provides a key point understanding method, including: obtaining an image to be detected, and obtaining position prompt information of target key points; performing semantic understanding on the image to be detected based on the position prompt information to obtain semantic information of the target key points in the image to be detected, and the semantic information is used to determine the position information of the target key points in the image to be detected.

[0006] Therefore, by combining the position prompt information related to the target key points with the image to be detected, the semantic understanding of the target key points in the image to be detected can be achieved, and the obtained semantic information can be used to locate the target key points to realize the detection of the key points in the image.

[0007] The method further includes: obtaining position information of target key points in the image to be detected based on semantic information.

[0008] Therefore, the semantics of the key points are preliminarily identified, and the positions of the key points are determined based on the semantic understanding to achieve key point detection in the image.

[0009] Among them, based on the position prompt information, semantic understanding of the image to be detected is performed to obtain the semantic information of the target key points in the image to be detected, including: encoding based on the image to be detected and the position prompt information to obtain encoding features; decoding the encoding features to obtain semantic text of the target key points in the image to be detected, wherein the semantic text includes a position mark indicating the position of the key point; based on the semantic information, obtaining the position information of the target key points in the image to be detected includes: obtaining a sub-coding feature of the decoded position mark; based on the sub-coding feature, obtaining the position information of the target key points in the image to be detected.

[0010] Therefore, the position mark is associated with the position of the key point, and the position information of the key point can be obtained by using the sub-coding features that can be decoded to obtain the position mark, thereby realizing the key point detection in the image.

[0011] Wherein, obtaining the position information of the target key point in the image to be detected based on the sub-coding feature includes: performing regression using the sub-coding feature to obtain the coordinates of the target key point in the image to be detected.

[0012] Therefore, the sub-coding features can be decoded to obtain the position mark, and the sub-coding features can be used for regression to obtain the key point coordinates to achieve key point detection.

[0013] Among them, encoding based on the image to be detected and the position prompt information to obtain the encoded feature includes: extracting the feature to be detected from the image to be detected; encoding based on the feature to be detected and the position prompt information to obtain the encoded feature.

[0014] Therefore, the features to be detected are extracted from the image to be detected, and then the features to be detected and the position prompt information are encoded to achieve the transformation of the image to be detected and the position prompt information, so as to be used for decoding to obtain semantic information and realize semantic understanding.

[0015] Among them, the position prompt information includes the support image corresponding to the image to be detected and the reference position information of the target key points in the support image; encoding is performed based on the features to be detected and the position prompt information to obtain the encoded features, including: feature extraction of the support image to obtain support image features, and feature extraction of the reference position information to obtain reference position features; fusing the support image features and the reference position features to obtain reference key point features; encoding is performed based on the features to be detected and the reference key point features to obtain the encoded features.

[0016] Therefore, the position hint information includes supporting images and reference position information. Through feature extraction and fusion coding, reference key point features related to the target key points can be obtained, which can be used to encode the features to be detected to achieve key point detection.

[0017] Among them, encoding is performed based on the features to be detected and the reference key point features to obtain the encoded features, including: embedding the reference key point features into preset text to obtain key point prompt features, wherein the preset text is used to instruct the large language model to obtain semantic information; and using the large language model to fuse the key point prompt features and the features to be detected to obtain the encoded features.

[0018] Therefore, the reference key point features are embedded in the text indicating the large language model, so that the key point prompt features and the features to be detected are processed by the large language model to obtain semantic information and realize semantic understanding.

[0019] Among them, the position prompt information includes descriptive text related to the target key points in the image to be detected and indicative text used to instruct the large language model to obtain semantic information; encoding is performed based on the features to be detected and the position prompt information to obtain the encoded features, including: using the large language model to fuse the features to be detected and the position prompt information to obtain the encoded features.

[0020] Therefore, the location prompt information includes a description text and an indication text indicating a large language model, so as to use the large language model to process the features to be detected and the location prompt information to obtain semantic information and realize semantic understanding.

[0021] Among them, semantic understanding of the image to be detected based on the position prompt information to obtain the semantic information of the target key points in the image to be detected is achieved by using a target model; the method also includes the following steps to perform target training on the target model: using the target model to semantically understand the sample image based on the sample position prompt information to obtain sample semantic information of the sample key points in the sample image; wherein the sample image is annotated with real semantic information; and adjusting the parameters of the target model based on at least the difference between the sample semantic information and the real semantic information.

[0022] Therefore, the target model is trained using sample images annotated with real semantic information to obtain a target model that can achieve semantic understanding.

[0023] Among them, the sample image is also marked with real position information, and the real position information includes the coordinates of the sample key points in the sample image. Before adjusting the parameters of the target model, the method also includes: using the target model to obtain the sample position information of the sample key points in the sample image based on the sample semantic information; at least based on the difference between the sample semantic information and the real semantic information, adjusting the parameters of the target model includes: adjusting the parameters of the target model based on the difference between the sample semantic information and the real semantic information, and the difference between the sample position information and the real position information.

[0024] Therefore, the sample images are also annotated with real position information, and the target model is trained using the sample images to obtain a target model that can realize key point detection.

[0025] The target model includes a large language model, the large language model includes a fine-tuning module, and during the target training process, the parameters of the fine-tuning module are used as objects of parameter adjustment.

[0026] Therefore, the large language model is equipped with a fine-tuning module, which can efficiently adjust the parameters of the large language model and improve training efficiency.

[0027] The real semantic information includes a position mark and at least one of a real category of the target to which the sample key point belongs and a real name of the sample key point.

[0028] Therefore, the sample images are annotated with location marks, the true category and real name of the target to guide the target model to correctly understand the semantics of key points and improve the accuracy of semantic understanding.

[0029] The present application provides a model training method, comprising: obtaining a sample image, and obtaining sample position prompt information of sample key points, wherein the sample image is annotated with real semantic information; using a target model to perform semantic understanding on the sample image based on the sample position prompt information, to obtain sample semantic information of the sample key points in the sample image, wherein the sample semantic information can be used to determine the sample position information of the sample key points in the sample image; and adjusting the parameters of the target model based on at least the difference between the sample semantic information and the real semantic information.

[0030] Therefore, the target model is trained using sample images annotated with real semantic information to obtain a target model that can achieve semantic understanding. The obtained sample semantic information can be used to locate sample key points to achieve key point detection in the image.

[0031] The present application provides a key point understanding device, including an acquisition module and a semantic understanding module. The acquisition module is used to acquire an image to be detected, and to acquire position prompt information of a target key point; the semantic understanding module is used to perform semantic understanding on the image to be detected based on the position prompt information, and obtain the semantic information of the target key point in the image to be detected. The semantic information can be used to determine the position information of the target key point in the image to be detected.

[0032] The present application provides a model training device, including an acquisition module, a model processing module and an adjustment module, wherein the acquisition module is used to acquire a sample image and obtain sample position prompt information of a sample key point, wherein the sample image is annotated with real semantic information; the model processing module is used to use a target model to perform semantic understanding on the sample image based on the sample position prompt information, and obtain sample semantic information of the sample key point in the sample image, and the sample semantic information can be used to determine the sample position information of the sample key point in the sample image; the adjustment module is used to adjust the parameters of the target model based on at least the difference between the sample semantic information and the real semantic information.

[0033] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement any of the above-mentioned key point understanding methods or model training methods.

[0034] The present application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement any of the above-mentioned key point understanding methods or model training methods.

[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0037] Figure 1 This is a flowchart of an embodiment of a method for understanding the key points of this application;

[0038] Figure 2 It is a flowchart of another embodiment of the key point understanding method of the present application;

[0039] Figure 3 is a flowchart of another embodiment of step S230 of the present application;

[0040] Figure 4 is a flowchart of another embodiment of step S331 of the present application;

[0041] Figure 5 It is a schematic diagram of another embodiment of the key point understanding method of the present application;

[0042] Figure 6 It is a schematic diagram of another embodiment of the key point understanding method of the present application;

[0043] Figure 7 It is a flowchart of an embodiment of the model training method of the present application;

[0044] Figure 8 It is a schematic diagram of the framework of an embodiment of a key point understanding device of the present application;

[0045] Fig. 9 It is a schematic diagram of the framework of an embodiment of the model training device of the present application;

[0046] Fig.10 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0047] Fig.11 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0048] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0049] In the following description, for the purpose of explanation rather than limitation, specific details such as specific subsystem structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0050] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0051] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of a method for understanding the key points of the present application.

[0052] Specifically, the method may include:

[0053] Step S110: Acquire the image to be detected.

[0054] Step S120: Obtaining location prompt information of target key points.

[0055] It should be noted that the execution order of step S110 and step S120 is not limited and can be set according to actual application requirements.

[0056] The key point understanding method provided in the present application can be used to understand the semantics of key points in an image, wherein the target key points can be used as key points to be semantically understood, and the image to be detected can be an image containing the target key points.

[0057] The position prompt information is related to the position information of the target key point, and can be used to prompt the position of the target key point in the image to be detected, so as to understand the semantics of the target key point in the image to be detected.

[0058] Step S130: semantically understand the image to be detected based on the position prompt information to obtain semantic information of target key points in the image to be detected.

[0059] The form of the semantic information can be set according to the actual application needs, for example, it can be in text form, or it can also be in other forms such as voice. The semantic information can represent the meaning of the target key point in the image to be detected, for example, the category of the target to which the target key point belongs, the name of the target key point, the position-related information of the target key point, etc.

[0060] In a specific application scenario, the semantic information of the target key point in the image to be detected may be in text form, specifically “target category: cat”.

[0061] It should be noted that the semantic information of the target key points can be used to determine the location information of the target key points to detect the target key points. In the process of key point detection, the semantics of the key points are first understood, and then the location of the key points is determined based on the semantic understanding. The model's understanding of the semantics of the key points is conducive to more accurate determination of the location of the key points and improves the accuracy of key point detection.

[0062] See also Figure 2 , Figure 2 1 is a flow chart of another embodiment of the key point understanding method of the present application. Specifically, the method may include:

[0063] Step S210: Acquire the image to be detected.

[0064] Step S220: Obtaining location prompt information of target key points.

[0065] Step S230: semantically understand the image to be detected based on the position prompt information to obtain semantic information of target key points in the image to be detected.

[0066] Steps S210 to S230 may refer to the relevant contents in the above-mentioned embodiments, and will not be described in detail here. After obtaining the semantic information of the target key point in the image to be detected, the position of the key point can be further located based on understanding the semantics of the key point.

[0067] Step S240: Based on the semantic information, obtain the position information of the target key point in the image to be detected.

[0068] In the above scheme, a key point detection paradigm based on thought chain is proposed, which first preliminarily identifies the semantics of the key points, and then accurately determines the location of the key points based on language understanding. The model's analysis and understanding of the semantics of key points is conducive to more accurately determining the location of key points and improving the accuracy of key point detection.

[0069] See also Figure 3 , Figure 3 2 is a flow chart of another embodiment of step S230 of the present application. Specifically, step S230 may include:

[0070] Step S331: Encode based on the image to be detected and the position prompt information to obtain a coding feature.

[0071] The position prompt information may be in various forms, which may be set according to actual application requirements. For example, the reference position information of the target key point in the image may be used as the position prompt information, or the position prompt information may be in text form.

[0072] It should be noted that different forms of location prompt information may have corresponding encoding processes.

[0073] In some embodiments, the encoding process can be implemented using a large language model.

[0074] In some embodiments, the encoding process can be implemented using a large language model and other encoding modules.

[0075] Step S332: Decode the encoded features to obtain the semantic text of the target key points in the image to be detected.

[0076] In this embodiment, semantic information in text form is taken as an example for explanation. Semantic text is obtained after encoding and decoding the image to be detected and the position prompt information as semantic information to achieve semantic understanding of key points in the image.

[0077] In a specific application scenario, the semantic text may include "object name": panda; "keypoint name": left ear.

[0078] In the above solution, a large language model is used to understand the semantics of key points and represent them more accurately, thereby improving the accuracy of subsequent key point detection.

[0079] In some embodiments, the semantic text may include position markers indicating the locations of key points.

[0080] In a specific application scenario, the semantic text may include "object name": panda; "keypoint name": left ear"; "keypoint position": <keypoint>.in, <keypoint>As a location marker.

[0081] In some embodiments, after obtaining the semantic text containing the position mark, since the semantic text is decoded as a coding feature, a sub-coding feature that can be decoded to obtain the position mark is obtained, and based on the sub-coding feature of the decoded position mark, the position information of the target key point in the image to be detected is obtained.

[0082] In a specific application scenario, semantic text may include "keypoint position": <keypoint>In this application, the location information of key points (such as coordinate values) is not obtained directly in the form of digital text, but a special token is introduced in the vocabulary <keypoint>, as a position marker indicating the position of the key point. The text decoded by the model contains the position marker. After decoding, the sub-encoding feature used to decode the position marker can be obtained from the features generated in the encoding and decoding process. The sub-encoding feature is related to the position of the target key point in the image to be detected and can be used to obtain the position information.

[0083] Furthermore, obtaining the position information of the target key point in the image to be detected based on the sub-coding feature may include: performing regression using the sub-coding feature to obtain the coordinates of the target key point in the image to be detected, thereby achieving pixel-level key point detection.

[0084] In a specific application scenario, the feedforward neural network (FFN) can be used to predict the head from <keypoint>The output potential embedding u kpt Calculate the 2D keypoint position y:

[0085] y=FFN(u kpt )∈R 2

[0086] See also Figure 4 , Figure 4 2 is a flow chart of another embodiment of step S331 of the present application. Specifically, step S331 may include:

[0087] Step S4311: extract the features to be detected from the image to be detected.

[0088] It should be noted that the specific method of extracting the feature to be detected can be set according to the actual application needs. The feature to be detected can represent the feature of the image to be detected, which is used to obtain semantic information in combination with the position prompt information.

[0089] In some embodiments, feature extraction may be performed on the image to be detected to directly obtain the features to be detected.

[0090] In some embodiments, feature extraction may be performed on the image to be detected, and the extracted features may be transformed to obtain the features to be detected.

[0091] In one implementation scenario, feature extraction is performed on the image to be detected to obtain spatial features, the two-dimensional space is divided into a number of sub-regions, and the spatial features in each sub-region are transformed to obtain sub-features of each sub-region as features to be detected. The transformation operation may include pooling, linear projection, and flattening.

[0092] In a specific application scenario, the spatial feature of each sub-region is represented by F q , there are p sub-regions in total, and adaptive average pooling is applied to each sub-region, and then linear projection is performed and flattened into a one-dimensional representation to obtain the sub-features of each sub-region, and finally the features to be detected are obtained The details are as follows:

[0093] z q =Linear(AvgPool(F q ))

[0094] Step S4312: Encode based on the features to be detected and the position prompt information to obtain encoded features.

[0095] In some embodiments, a large language model may be used to participate in the encoding process of the image to be detected and the position prompt information.

[0096] In some embodiments, in addition to processing the image to be detected to obtain the features to be detected, the position prompt information may also be processed to obtain reference key point features, and encoding may be performed based on the features to be detected and the reference key point features to obtain encoded features.

[0097] It should be noted that, according to the form of the location prompt information, corresponding processing can be performed. The specific processing steps can be set according to actual application needs.

[0098] In some embodiments, the position prompt information may include a supporting image corresponding to the image to be detected and reference position information of target key points in the supporting image.

[0099] In one implementation scenario, feature extraction is performed on a supporting image to obtain supporting image features, and feature extraction is performed on reference position information to obtain reference position features, and the supporting image features and reference position features are fused to obtain reference key point features.

[0100] In one implementation scenario, the reference position information may be the coordinates of key points of the target in the supporting image.

[0101] In a specific application scenario, position embedding (PE) can be used to encode the coordinates of the target key points in the support image, such as sine-cosine position embedding, and then the multi-layer perceptron (MLP) is used to increase the dimension of the above encoding results to obtain reference position features, and adjust the coordinates to align with the image feature space F. The specific formula is as follows:

[0102] F p =MLP(PE(x))

[0103] Among them, F p Represents the reference position feature, the two-dimensional coordinates of the key point x∈R 2 .

[0104] In a specific application scenario, the supporting image may be consistent with the image to be detected, so that the semantic information of the target key point at a certain position in the image to be detected can be obtained.

[0105] In a specific application scenario, the supporting image may be different from the image to be detected, and both the supporting image and the image to be detected contain targets of the same category, and the target key points are key points belonging to the target category. For example, both the supporting image and the image to be detected contain a panda, and both contain a left ear as a target key point.

[0106] In some embodiments, the fusion of the support image features and the reference image features can be implemented using a cross-attention mechanism (CrossAttnLayers). p As a query, support image features F s as keys and values ​​to extract location-specific visual features indicated by the cue:

[0107] z p =CrossAttnLayers(F p ,F s )

[0108] Among them, z p Represents location-specific visual features, i.e., reference keypoint features.

[0109] In some embodiments, encoding is performed based on the features to be detected and the reference key point features to obtain the encoded features, which can be achieved by using a large language model. Specifically, the reference key point features are embedded in a preset text to obtain key point prompt features, and the key point prompt features and the features to be detected are used as inputs of the large language model. The preset text is used to instruct the large language model to obtain semantic information.

[0110] In a specific application scenario, the preset text is "What is the keypoint <keypoint>name of the object in the image?Please also accurately locate the position of the keypoint.”。Among them, in <keypoint>The reference key point features are embedded at the locations to obtain the key point prompt features. The key point prompt features and the features to be detected are fused and encoded using a large language model to obtain the encoded features, which are then decoded to obtain the semantic text.

[0111] In some embodiments, the position prompt information may include description text related to the target key point in the image to be detected and instruction text for instructing the large language model to obtain semantic information.

[0112] In some embodiments, a large language model may be used to implement encoding based on the feature to be detected and the position prompt information to obtain the encoded feature. The position prompt information includes a description text and an instruction text, and the position prompt information and the feature to be detected are used as inputs of the large language model.

[0113] In a specific application scenario, the position prompt information may include descriptive text related to the target key points in the image to be detected and instructive text for instructing the large language model to obtain semantic information. The large language model is used to fuse the features to be detected and the position prompt information to obtain the encoded features, and the encoded features are decoded to obtain the semantic text.

[0114] See also Figure 5 , Figure 5 It is a schematic diagram of another embodiment of the key point understanding method of the present application.

[0115] The key point understanding method in this application can be implemented using a target model. Figure 5 Schematic diagram of the model structure and processing process of the target model.

[0116] The target model includes a visual encoder, a linear module and a large language model. The visual encoder can be used to extract image-level spatial features, for example, to extract features of the image to be detected. The linear module can be used to transform the features of the sub-region to obtain the features to be detected. The large language model is used to obtain the semantic information of the target key points in the image to be detected based on the input information.

[0117] In some embodiments, the target model may include a cue encoder and a cue feature extractor, which may process position cue information in the form of images.

[0118] In some embodiments, the target model does not include a prompt encoder and a prompt feature extractor, and the target model can only support position prompt information in text form.

[0119] In some embodiments, the target model may not include a feedforward neural network, and the target model is only used for semantic understanding of key points.

[0120] In some embodiments, the target model may include a feedforward neural network, and the target model may be used for semantic understanding of key points or for key point detection. Both semantic information and key point location information may be output as a model.

[0121] In the above scheme, the model can process inputs of multiple modalities, flexibly adapt to different application scenarios, and realize semantic understanding and position judgment of key points.

[0122] It should be noted that if Figure 5 As shown in the figure, during the model fine-tuning process, the modules marked with triangle symbols are used as parameter adjustment objects, including the visual encoder, the prompt encoder, the linear module, the prompt feature extractor, the fine-tuning module, and the feedforward neural network. The modules without triangle symbols are not used as parameter adjustment objects, such as the pre-trained large language model except the fine-tuning module in the figure.

[0123] It is understandable that the target model can be used to perform tasks related to key points, including key point semantic understanding and key point detection. For example, the target key points in the image to be detected can be semantically understood, and the position information of the target key points in the image to be detected can also be detected.

[0124] In a specific application scenario, the position hint information in the form of an image can be selected. Specifically, it can include the support image and the reference position information of the target key points in the support image. In some cases, the support image and the image to be detected can be the same image to achieve semantic understanding of the target key points in the image to be detected. In this case, the type of input information of the model is as follows: Figure 5 As shown in ①, it includes query image, support image and support keypoint prompt, which is different from Figure 5 In the example in , the query image and the support image are the same image. In some cases, the support image and the image to be detected can contain objects of the same category to understand the semantic key points of the target in the image to be detected. In this case, the input information of the model is as follows Figure 5 As shown in ②, it includes query image, support image and support keypoint prompt.

[0125] It is understandable that the method provided in the present application does not limit the category of key points, and providing supporting images containing objects of the same category can enable semantic understanding and detection of key points of objects of that category.

[0126] In a specific application scenario, you can choose text-based location prompt information. In this case, the model input information is as follows: Figure 5 As shown in ③, it includes a query image and location prompt information in text form. The target model includes a feedforward neural network for obtaining the location information of the target key point based on the semantic information, and at least one of the semantic information and the location information can be used as the output of the model.

[0127] In a specific application scenario, given an image input I∈R H×W×3 , use the visual encoder V to extract image-level spatial features F = V(I)∈R h×w×d , where d represents the feature dimension, and h and w are obtained by downsampling H and W.

[0128] The two-dimensional spatial grid of features is divided into p local regions, and the feature of each local region is represented as F q Adaptive average pooling is applied to each local region, followed by linear projection and flattening into a one-dimensional representation: q =Linear(AvgPool(F q )) Obtained query visual token can be input into a large language model, where d1 represents the input dimension of the large language model.

[0129] If the support image is used as the location hint information, the spatial feature of the support image is expressed as F s , these features are not directly input into the large language model, but are processed by the prompt feature extractor to obtain prompt-oriented features. In addition to processing images, we introduce the two-dimensional coordinates x∈R 2 The keypoint hints are composed of coordinates describing the keypoint locations in the image. A hint encoder is introduced to adjust this hint input to align with the image feature space F. The hint encoder encodes the keypoint coordinates using sine-cosine position embedding (PE), which is then processed by a multi-layer perceptron (MLP): F p =MLP(PE(x)).

[0130] The cue feature extractor aims to extract cue-related specific features from image features. Figure 5 As shown in Figure 2, the semantics of the keypoint hints directly correspond to the support image. The hint feature extractor is instantiated using a two-layer transformer with a cross-attention mechanism (CrossAttnLayers). This mechanism uses F p As a query, use F s as keys and values ​​to extract location-specific visual features indicated by the cue: z p =CrossAttnLayers(F p ,F s ).

[0131] Among them, z p Represents position-specific visual features that can be used to embed preset text. Essentially, the hint feature extractor is able to incorporate global image features to enhance keypoint identification. This is particularly beneficial for distinguishing mirror-symmetric keypoints (such as left and right eyes), which can be very ambiguous when relying solely on local image features.

[0132] Use the pre-trained large language model to decode the task-related text response, whose input includes the query visual token z from the visual encoder q , and also includes the hint token z from the hint feature extractor p and at least one of task-specific textual user instructions. On the other hand, for the decoding of keypoint locations. We introduce a special token in the vocabulary W <keypoint>, instead of obtaining key points directly in digital text form. A feed-forward neural network (FFN) prediction head can be used to embed the output potential u from a special token kpt Calculate the 2D keypoint position y:

[0133] y=FFN(u kpt )∈R 2

[0134] It should be noted that the output potential embedding u kpt To decode the encoded features of a particular token.

[0135] See also Figure 6 , Figure 6 Schematic diagram of another embodiment of the key point understanding method of the present application. The key point understanding method of the present application can be implemented using a target model. Before using the target model, the method can also include the step of performing target training on the target model:

[0136] Step S610: using the target model to perform semantic understanding on the sample image based on the sample position prompt information, and obtaining sample semantic information of the sample key points in the sample image.

[0137] It should be noted that the specific steps performed using the target model may refer to the relevant contents in the aforementioned embodiments, which will not be described in detail here.

[0138] Among them, the sample images are annotated with real semantic information.

[0139] In one implementation scenario, the actual semantic information includes a location tag.

[0140] In one implementation scenario, the true semantic information includes at least one of the true category of the target to which the sample key point belongs and the true name of the sample key point.

[0141] Step S620: Adjust the parameters of the target model based at least on the difference between the sample semantic information and the true semantic information.

[0142] In one implementation scenario, for the key point semantic understanding task, the focus is on extracting the semantic information of the key points. The training target is the language modeling loss. The cross entropy loss can be used to calculate the vocabulary size W of the large language model segmenter:

[0143] L=L llm (w,w′)

[0144] Where w′ is the true text response.

[0145] In some embodiments, the target model is also used to obtain the position information of the target key points. Accordingly, the sample image is also annotated with the real position information, and specifically, the real position information can be the coordinates of the sample key points in the sample image.

[0146] In some embodiments, the method may further include obtaining sample position information of sample key points in the sample image based on the sample semantic information using the target model, and adjusting the parameters of the target model according to the difference between the sample semantic information and the real semantic information, and the difference between the sample position information and the real position information.

[0147] In one implementation scenario, for the key point detection task, the loss can be obtained using the following formula:

[0148] L=λ∥yy ′ ∥+L llm (w,w′)

[0149] Where y′ is the true keypoint position, λ is the loss weight used to balance the learning of keypoint regression and text generation, and illustratively, λ=2.

[0150] Please continue reading Figure 5 The target model includes a large language model, and the large language model includes a fine-tuning module, wherein the large language model can be any pre-trained large language model. During the target training process, for the large language model, the parameters of the fine-tuning module can be used as the object of parameter adjustment, and the parameters other than the fine-tuning module can be not adjusted. The parameters of other modules of the target model are adjusted normally during the target training process.

[0151] In a specific application scenario, the large language model includes a LoRA module, and the parameters of the LoRA module are used as objects of parameter adjustment during the target training process.

[0152] See also Figure 7 , Figure 7 It is a flow chart of an embodiment of the model training method of the present application.

[0153] Specifically, the method includes:

[0154] Step S710: Acquire a sample image.

[0155] Step S720: Obtain sample position prompt information of sample key points.

[0156] It should be noted that the execution order of step S710 and step S720 is not limited.

[0157] Step S730: Using the target model to perform semantic understanding on the sample image based on the sample position prompt information, and obtaining sample semantic information of the sample key points in the sample image.

[0158] Step S740: Adjust the parameters of the target model based at least on the difference between the sample semantic information and the true semantic information.

[0159] For the specific steps, please refer to the relevant contents in the aforementioned embodiments, which will not be described in detail here.

[0160] In one implementation scenario, the actual semantic information includes a location tag.

[0161] In one implementation scenario, the true semantic information includes at least one of the true category of the target to which the sample key point belongs and the true name of the sample key point.

[0162] In some embodiments, the target model can also be used to detect the position information of the key points, and the sample image can also be annotated with the real position information. The method can also include: using the target model to determine the sample position information of the sample key points in the sample image based on the sample semantic information. When adjusting the model parameters, the difference between the sample semantic information and the real semantic information, and the difference between the sample position information and the real position information can be combined.

[0163] In the above scheme, the target model is trained, and the real semantic information is used to guide the model to accurately understand the semantics of key points, and can improve the accuracy of subsequent key point detection of the model.

[0164] In some embodiments, a target model may be trained to perform different tasks. Alternatively, a target model may be trained for each task, and a target model may be selected for reasoning as needed. Multiple target models may have different structures.

[0165] See also Figure 8 , Figure 8 It is a framework diagram of an embodiment of a key point understanding device of the present application.

[0166] In this embodiment, the key point understanding device 80 includes an acquisition module 81 and a semantic understanding module 82. The acquisition module 81 is used to acquire the image to be detected and to acquire the position prompt information of the target key points; the semantic understanding module 82 is used to perform semantic understanding on the image to be detected based on the position prompt information to obtain the semantic information of the target key points in the image to be detected. The semantic information can be used to determine the position information of the target key points in the image to be detected.

[0167] See also Fig. 9 , Fig. 9 It is a framework diagram of an embodiment of the model training device of the present application.

[0168] In this embodiment, the model training device 90 includes an acquisition module 91, a model processing module 92 and an adjustment module 93. The acquisition module 91 is used to acquire a sample image and obtain sample position prompt information of sample key points, wherein the sample image is annotated with real semantic information; the model processing module 92 is used to use the target model to perform semantic understanding of the sample image based on the sample position prompt information, and obtain the sample semantic information of the sample key points in the sample image, and the sample semantic information can be used to determine the sample position information of the sample key points in the sample image; the adjustment module 93 is used to adjust the parameters of the target model based on at least the difference between the sample semantic information and the real semantic information.

[0169] See also Fig.10 , Fig.10 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application.

[0170] In this embodiment, the electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is used to execute the program instructions stored in the memory 101 to implement the steps in any of the above key point understanding methods or model training method embodiments. In a specific implementation scenario, the electronic device 100 may include, but is not limited to: computer equipment, electrical equipment, microcomputers, desktop computers, servers, and in addition, the electronic device 100 may also include mobile devices such as laptops and tablet computers, which are not limited here.

[0171] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned key point understanding methods or model training method embodiments. The processor 102 can also be referred to as a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip having signal processing capabilities. The processor 102 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 102 may be implemented by an integrated circuit chip.

[0172] See also Fig.11 , Fig.11 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application.

[0173] The computer-readable storage medium 110 provided in this embodiment stores program instructions 111 that can be executed by a processor. When the program instructions 111 are executed by the processor, they are used to implement any of the above-mentioned key point understanding methods or model training methods.

[0174] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another subsystem, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0176] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.< / keypoint> < / keypoint> < / keypoint> < / keypoint> < / keypoint> < / keypoint> < / keypoint> < / keypoint>

Claims

1. A key point understanding method, characterized in that: The method comprises: Obtain the image to be detected, and obtain the location prompt information of the target key points; Based on the position prompt information, semantic understanding is performed on the image to be detected to obtain semantic information of the target key point in the image to be detected, and the semantic information is used to determine the position information of the target key point in the image to be detected.

2. The method according to claim 1, characterized in that: The method further comprises: Based on the semantic information, the position information of the target key point in the image to be detected is obtained.

3. The method according to claim 2, characterized in that The performing semantic understanding on the image to be detected based on the position prompt information to obtain semantic information of the target key point in the image to be detected includes: Encoding is performed based on the image to be detected and the position prompt information to obtain a coding feature; Decoding the encoded features to obtain semantic text of the target key point in the image to be detected, wherein the semantic text includes a position mark indicating the position of the key point; The obtaining, based on the semantic information, the position information of the target key point in the image to be detected comprises: Obtaining a sub-coding feature of the position mark obtained by decoding; Based on the sub-coding features, the position information of the target key points in the image to be detected is obtained.

4. The method according to claim 3, characterized in that The obtaining, based on the sub-coding feature, the position information of the target key point in the image to be detected comprises: The sub-coding features are used to perform regression to obtain the coordinates of the target key points in the image to be detected.

5. The method according to claim 3 or 4, characterized in that: The encoding based on the image to be detected and the position prompt information to obtain the encoding features includes: Extracting features to be detected from the image to be detected; Encoding is performed based on the feature to be detected and the position prompt information to obtain the encoded feature.

6. The method according to claim 5, characterized in that The position prompt information includes a support image corresponding to the image to be detected and reference position information of the target key point in the support image; the encoding based on the feature to be detected and the position prompt information to obtain the encoded feature includes: Performing feature extraction on the supporting image to obtain supporting image features, and performing feature extraction on the reference position information to obtain reference position features; Fusing the supporting image features and the reference position features to obtain reference key point features; Encoding is performed based on the feature to be detected and the reference key point feature to obtain the encoded feature.

7. The method according to claim 6, characterized in that The encoding based on the feature to be detected and the reference key point feature to obtain the encoded feature includes: Embedding the reference key point feature into a preset text to obtain a key point prompt feature, wherein the preset text is used to instruct a large language model to obtain the semantic information; The key point prompt feature and the feature to be detected are fused and encoded using the large language model to obtain the encoded feature.

8. The method according to claim 5, characterized in that The position prompt information includes a description text related to the target key point in the image to be detected and an instruction text for instructing a large language model to obtain the semantic information; the encoding based on the feature to be detected and the position prompt information to obtain the encoded feature includes: The large language model is used to perform fusion encoding on the feature to be detected and the position prompt information to obtain the encoded feature.

9. The method according to any one of claims 1 to 8, characterized in that The semantic understanding of the image to be detected based on the position prompt information to obtain the semantic information of the target key point in the image to be detected is achieved by using a target model; the method also includes the following steps to perform target training on the target model: Using the target model to perform semantic understanding on the sample image based on the sample position prompt information, to obtain sample semantic information of the sample key points in the sample image; wherein the sample image is annotated with real semantic information; Based at least on the difference between the sample semantic information and the true semantic information, the parameters of the target model are adjusted.

10. The method according to claim 9, characterized in that The sample image is also annotated with real position information, and the real position information includes the coordinates of the sample key points in the sample image. Before adjusting the parameters of the target model, the method further includes: Obtaining sample position information of the sample key points in the sample image using the target model based on the sample semantic information; The adjusting the parameters of the target model based at least on the difference between the sample semantic information and the real semantic information includes: adjusting the parameters of the target model based on the difference between the sample semantic information and the true semantic information, and the difference between the sample position information and the true position information; and / or, The target model includes a large language model, the large language model includes a fine-tuning module, and during the target training process, parameters of the fine-tuning module are used as objects of parameter adjustment; and / or, The real semantic information includes a position mark, and at least one of a real category of the target to which the sample key point belongs and a real name of the sample key point.

11. A model training method, characterized in that: The method comprises: Acquire a sample image, and acquire sample position prompt information of a sample key point, wherein the sample image is annotated with real semantic information; Using the target model to perform semantic understanding on the sample image based on the sample position prompt information, to obtain sample semantic information of the sample key points in the sample image, wherein the sample semantic information can be used to determine sample position information of the sample key points in the sample image; Based at least on the difference between the sample semantic information and the true semantic information, the parameters of the target model are adjusted.

12. A key point understanding device, characterized in that: The device comprises: An acquisition module is used to acquire the image to be detected and obtain the location prompt information of the target key points; A semantic understanding module is used to perform semantic understanding on the image to be detected based on the position prompt information to obtain semantic information of the target key points in the image to be detected, and the semantic information can be used to determine the position information of the target key points in the image to be detected.

13. A model training device, characterized in that: The device comprises: An acquisition module, used to acquire a sample image and acquire sample position prompt information of a sample key point, wherein the sample image is annotated with real semantic information; A model processing module, used to perform semantic understanding on the sample image based on the sample position prompt information using a target model, and obtain sample semantic information of the sample key points in the sample image, wherein the sample semantic information can be used to determine sample position information of the sample key points in the sample image; An adjustment module is used to adjust the parameters of the target model based on at least the difference between the sample semantic information and the true semantic information.

14. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores program instructions, and when the program instructions are executed by the processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Citation Information

Patent Citations

  • Model training method and device, detection method and device, equipment and storage medium

    CN115578623A

  • Target detection method and device and computer equipment

    CN116758277A

  • Image processing method and related equipment

    CN117314944A

  • Electric meter box appearance anomaly detection method and device based on image registration and change detection

    CN117974726A

  • Target key point detection method and system

    CN118247519A