Image determination and model training method, apparatus, device, medium, and program product

Abstract

This application provides an image determination and model training method, apparatus, device, medium, and program product; the method includes: extracting first language features from retrieved text; determining a matching result between the first language features and a bird's-eye view feature set corresponding to an image set; and determining a target image from the image set based on the matching result. This application can meet the image retrieval needs in diverse scenarios, including complex traffic environments.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image determination and model training method, apparatus, device, medium, and program product. Background Technology

[0002] In practical applications, retrieval techniques for local planar images have been successfully applied to various scenarios. However, local planar images cannot capture the global features of diverse environments, including complex traffic environments. Therefore, retrieval techniques for local planar images in related technologies cannot meet the image retrieval needs of diverse scenarios, including complex traffic environments. Summary of the Invention

[0003] Based on the above technical issues, this application provides an image determination and model training method, apparatus, device, medium, and program product that can meet the image retrieval needs in diverse scenarios, including complex traffic environments.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application first provides an image determination method, the method comprising:

[0006] Extracting first language features from the retrieved text;

[0007] Determine the matching result between the first language features and the bird's-eye view feature set corresponding to the image set;

[0008] Based on the matching results, the target image is determined from the image set.

[0009] This application embodiment also provides a model training method, the method comprising:

[0010] The initial retrieval model is trained based on sample data to obtain a retrieval model; wherein, the retrieval model is used to determine the matching result between the first language features and the bird's-eye view feature set corresponding to the image set; the first language features are extracted from the retrieval text; the matching result is used to determine the target image from the image set.

[0011] This application embodiment also provides an image determining device, the image determining device comprising:

[0012] The processing module is used to extract first language features from the retrieved text;

[0013] The determination module is used to determine the matching result between the first language features and the bird's-eye view features corresponding to the image set, and to determine the target image from the image set based on the matching result.

[0014] This application embodiment also provides a model training device, the model training device comprising:

[0015] The training module is used to train an initial retrieval model based on sample data to obtain a retrieval model; wherein, the retrieval model is used to determine the matching result between a first language feature and a bird's-eye view feature set corresponding to the image set; the first language feature is extracted from the retrieval text; the matching result is used to determine the target image from the image set.

[0016] This application also provides an electronic device, which includes a processor and a memory; the memory stores a computer program; when the computer program is executed by the processor, it can implement the image determination method or model training method as described above.

[0017] This application also provides a computer-readable storage medium storing a computer program; when the computer program is executed by a processor of an electronic device, it can implement the image determination method or model training method as described above.

[0018] This application also provides a computer program product, which includes a computer program; when the computer program is executed by the processor of an electronic device, it can implement the image determination method or model training method as described above.

[0019] The embodiments of this application have the following beneficial effects:

[0020] The image determination method provided in this application extracts first language features from the retrieved text and determines the matching result between the first language features and the bird's-eye view feature set corresponding to the image set. Through the above operations, it realizes the traversal matching of the bird's-eye view feature set corresponding to the image set based on the first language features corresponding to the retrieved text. Furthermore, the matching result can intuitively and accurately represent the degree of matching between the images in the image set and the retrieved text corresponding to the first language features. On this basis, the target image is determined from the image set based on the matching result, which improves the correlation between the target image and the retrieved text and realizes targeted search and filtering of images in the image set. On the other hand, since the matching result is determined based on the first language features and the bird's-eye view feature set corresponding to the image set, when the retrieved text and the image set correspond to diverse and complex scenes, the technical solution provided in this application can meet the filtering and retrieval needs of image sets corresponding to diverse and complex scenes. When the technical solution provided in this application is applied to an autonomous driving scenario, if the search text corresponds to an autonomous driving scenario and the image set corresponds to a BEV image in a complex traffic scenario, the above method can achieve accurate and flexible retrieval of BEV images corresponding to complex traffic scenarios based on the complex descriptive data corresponding to the autonomous driving scenario, thereby meeting the image retrieval needs for diverse scenarios including complex traffic environments. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating the effect of image retrieval comparison;

[0022] Figure 2 A schematic flowchart illustrating the image determination method provided in this application embodiment;

[0023] Figure 3A This is a schematic diagram of the structure of the knowledge graph provided in an embodiment of this application;

[0024] Figure 3B A schematic diagram illustrating the process of obtaining first language features provided in an embodiment of this application;

[0025] Figure 3C This is a schematic diagram of the retrieval model provided in the embodiments of this application;

[0026] Figure 4 A schematic flowchart illustrating the model training method provided in this application embodiment;

[0027] Figure 5A A comparative illustration of the different levels of sample description provided in the embodiments of this application;

[0028] Figure 5B A schematic diagram illustrating the comparison effect of retrieving two-dimensional images provided in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the structure of the image determination device provided in the embodiments of this application;

[0030] Figure 7 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0031] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0032] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0036] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0037] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0038] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0039] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0040] 1) Bird's Eye View (BEV): BEV has seen rapid development in the field of autonomous driving. This technology can accurately acquire the position and semantic information of objects in a planar environment, which is particularly important for tasks such as object detection and map semantic segmentation. BEV maps integrate data from different sensors into a consistent format that matches the planar geometry of the scene, thereby enhancing the two-dimensional projection effect.

[0041] 2) Large Language Model (LLM): This refers to a deep learning model trained on a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0042] 3) Knowledge Graph (KG): Also known as knowledge domain visualization or knowledge domain mapping map, it is a series of different graphics that show the development process and structural relationships of knowledge. It uses visualization technology to describe knowledge resources and their carriers, and to mine, analyze, construct, draw and display knowledge and the interrelationships between them.

[0043] With the rise of language and visual deep models, cross-modal image-text retrieval (ITR) has made significant progress in the past few years, and ITR can achieve good retrieval results when handling simple retrieval tasks. Figure 1 This is a schematic diagram illustrating the effect of image retrieval comparison. For example... Figure 1 As shown, using ITR to retrieve the input text "a man is drinking wine", results can be obtained as follows: Figure 1 The image of a man drinking wine, as shown, demonstrates that ITR can achieve good image retrieval results when the input text is "a brown dog" or in slightly more complex forms. However, ITR cannot handle the highly challenging retrieval tasks in complex traffic environments within autonomous driving scenarios.

[0044] On the other hand, thanks to vehicle data collection capabilities, crowdsourced vehicles, and the rapid development of the autonomous driving industry, data for autonomous driving scenarios has transitioned from a scarce to an abundant phase. However, uniformly distributed data cannot meet the needs of optimizing autonomous driving scenarios. For example, if the goal of optimizing an autonomous driving scenario is to improve the autonomous driving performance of vehicles on rural roads, sufficient rural road data must be retrieved to fine-tune the autonomous driving control model. Therefore, data mining has become a fundamental framework for providing specialized optimization for autonomous driving models, and a well-designed retrieval method plays a crucial role in the closed-loop data-driven process of autonomous driving data.

[0045] In summary, the demand for professional, high-precision image retrieval capable of carrying global environmental features is constantly increasing for autonomous driving scenarios. However, images retrieved using existing image retrieval methods lack global feature representations, thus limiting their text retrieval capabilities in complex driving scenarios. For example, in... Figure 1 In the context of autonomous driving, when the input text is "Arriving at an intersection, someone is loading a car, someone is walking a dog, crossing the pedestrian crossing, many cars, several trucks and buses, is there a bus on the right? Yes," the complex traffic scene described in the input text presents a significant challenge to the retrieval models used in autonomous driving scenarios. This requires a comprehensive understanding of the global context of the traffic scene and the ability to comprehend complex and lengthy text. These requirements exceed the data processing capabilities of the aforementioned retrieval models. Therefore, image retrieval methods from related technologies can provide... Figure 1 The middle section shows a collection of two-dimensional images, but this collection of images clearly cannot represent the current driving scenario of the vehicle.

[0046] To address the above technical problems, this application provides an image determination method. The image determination method provided by this application, based on... Figure 1 When searching the text "At the intersection, some people are loading vehicles, some are walking their dogs, crossing the pedestrian crossing, there are many cars, several trucks and buses, is there a bus on the right? Yes" you can get the following results. Figure 1 The rightmost image shows a bird's-eye view with global 3D features, which can meet the image retrieval needs in diverse scenarios, including complex traffic environments.

[0047] Figure 2 This is a flowchart illustrating the image determination method provided in an embodiment of this application, as shown below. Figure 2 As shown, this method can be implemented through the following steps:

[0048] S201. Extract first language features from the retrieved text.

[0049] In one implementation, the search text can be preset or input by the user, and the search text includes descriptive data for at least one scene object from multiple dimensions; for example, the scene object may include objects with living characteristics, such as pedestrians and pets; for example, the scene object may also include objects without living characteristics, such as vehicles, traffic lights, zebra crossings and lane dividers.

[0050] In one implementation, the retrieved text may include descriptive data of at least one of the following: the number, actions, poses, sizes, and relative positional relationships of scene objects in at least one scene object.

[0051] In one implementation, the features in the first language features can be context-dependent, and the first language features include at least one state feature of a scene object; for example, the state feature may include at least one corresponding feature of the scene object's color, shape, position, size, posture, and action.

[0052] In one implementation, the first language features may include a set of context-related features of at least one scene object in the spatial and / or temporal dimensions.

[0053] In one implementation, the first language features can be extracted in the following way:

[0054] The association state between scene objects in the search text is analyzed by LLM to obtain the state analysis results. Then, features are extracted from the search text to obtain the feature extraction results. The features in the feature extraction results are then integrated according to the state analysis results, and the integrated set of features is determined as the first language features. The state analysis results can include whether scene objects are associated and the degree of association.

[0055] S202. Determine the matching result between the first language features and the bird's-eye view feature set corresponding to the image set.

[0056] In one implementation, the image set can be pre-built or continuously expanded according to actual image retrieval needs; for example, the image set may include multiple panoramic top-down images, and the panoramic top-down images may carry global three-dimensional spatial features corresponding to multiple scenes; wherein, the multiple scenes may include at least one of the scenes mentioned above, and the three-dimensional spatial features may include the features of multiple and / or multiple scene objects that may exist in multiple scenes in three-dimensional space.

[0057] In one implementation, the panoramic top-down image in the image set may include BEV images and / or bird's-eye view images; wherein, the BEV image can be obtained by integrating multiple planar images and / or depth images, and the bird's-eye view image may include a top-down image acquired by an image acquisition device set at a high altitude in a bird's-eye view posture, and the bird's-eye view image may be a depth image.

[0058] In one implementation, the bird's-eye view feature set can be obtained in the following way:

[0059] A bird's-eye view feature set is obtained by performing global feature extraction on images in an image set using a neural network with image feature extraction capabilities. In other words, the bird's-eye view features in the bird's-eye view feature set can include global, spatial and / or temporal features carried by the images in the image set.

[0060] In one implementation, the matching result may include whether the first language features match the features in the bird's-eye view feature set, the degree of matching, etc.

[0061] In one implementation, the matching result can be determined in the following way:

[0062] Calculate the matrix similarity between the language feature matrix corresponding to the first language feature and the nth bird's-eye view matrix corresponding to the nth feature in the bird's-eye view feature set, and determine the calculated matrix similarity as the matching result; where n can be an integer greater than or equal to 1.

[0063] S203. Based on the matching results, determine the target image from the image set.

[0064] In one implementation, the target image may include at least one panoramic top-down image, or the target image may include a collection of multiple planar images or depth images corresponding to the BEV image.

[0065] In one implementation, the target image can be obtained in any of the following ways:

[0066] If the matching result includes multiple matching degrees, the panoramic top-down view image in the image set that corresponds to the maximum matching degree among the multiple matching degrees is determined as the target image; if the matching result includes a single matching degree, and the matching degree is greater than the degree threshold, then the image in the image set that corresponds to the matching degree is determined as the target image.

[0067] As can be seen from the above, the image determination method provided in this application extracts first language features from the searched text and determines the matching result between the first language features and the bird's-eye view feature set corresponding to the image set. Through the above operations, it realizes the traversal matching of the bird's-eye view feature set corresponding to the image set based on the first language features corresponding to the searched text. Furthermore, through the matching result, it can intuitively and accurately represent the degree of matching between the images in the image set and the searched text corresponding to the first language features. On this basis, the target image is determined from the image set based on the matching result, which improves the correlation between the target image and the searched text and realizes targeted search and filtering of images in the image set. On the other hand, since the matching result is determined based on the first language features and the bird's-eye view feature set corresponding to the image set, when the searched text and the image set correspond to diverse and complex scenes, the technical solution provided in this application can meet the filtering and retrieval needs of image sets corresponding to diverse and complex scenes. When the technical solution provided in this application is applied to an autonomous driving scenario, if the search text corresponds to an autonomous driving scenario and the image set corresponds to a BEV image in a complex traffic scenario, the above method can achieve accurate and flexible retrieval of BEV images corresponding to complex traffic scenarios based on the complex descriptive data corresponding to the autonomous driving scenario, thereby meeting the image retrieval needs for diverse scenarios including complex traffic environments.

[0068] Based on the foregoing embodiments, the image determination method provided in this application, which extracts first language features from the retrieved text, can be achieved through the following steps:

[0069] SA1. Perform feature extraction processing on the retrieved text to obtain the first text features.

[0070] The retrieved text includes descriptive data from multiple dimensions targeting the target scenario.

[0071] In one implementation, the target scenario may include scenarios with a complexity level greater than or equal to a first threshold and a diversity level greater than or equal to a second threshold; for example, the target scenario may include a complex traffic scenario, such as an autonomous driving scenario in a traffic congestion state.

[0072] In one implementation, the first text feature may include a vectorized representation of the retrieved text; correspondingly, the first text feature can be obtained in the following manner:

[0073] The first text feature is obtained by extracting features from the retrieved text using a text feature extraction model.

[0074] SA2. Determine the second text features corresponding to the target scene based on the knowledge graph.

[0075] Among them, the knowledge graph is at least associated with the target scene.

[0076] In one implementation, associating a knowledge graph with a target scene may include at least one of the following: the knowledge graph contains scene objects that may appear in the target scene, the actions that each scene object may perform or the state it is in, and the relative relationships between each scene object.

[0077] Figure 3A This is a schematic diagram of the structure of the knowledge graph provided in an embodiment of this application. For example, the knowledge graph 301 can correspond to a complex traffic environment in an autonomous driving scenario.

[0078] For example, the knowledge graph 301 may include five scene objects, from the first object to the fifth object, and each object is associated with a scene, a car, a person, a moving object, and a walking object.

[0079] In practical applications, a knowledge graph (KG) is a multi-relation graph that stores factual knowledge from the real world. It is typically represented as G = {E, R, S}, where E represents the set of entities, R represents the set of relations, and S represents relational facts. The observed facts in G are stored as a set of triples: G = {(h, r, t)}, where each triple consists of a head entity h ∈ E, a tail entity t ∈ E, and the relation r ∈ R between them. For example, G could be <scene, contain, car>.

[0080] In this embodiment, the knowledge graph can be obtained by analyzing scene objects and their relationships within an autonomous driving scenario. Each node in the knowledge graph corresponds to a keyword related to the autonomous driving scenario, and the embeddings associated with these nodes capture the relational representation of the autonomous driving keywords.

[0081] Furthermore, a knowledge graph can be obtained by simplifying the large-scale data and complex graph structure contained in the original knowledge graph of the autonomous driving field. For example, through a knowledge graph, entities in the autonomous driving field and their relationships can be represented in a low-dimensional space, while preserving their semantics in the original knowledge graph. For example, a modeling method based on semantic transformation distance can be used to construct the knowledge graph; specifically, a distance-based scoring function can be used to optimize the embedding, as shown in Equation (1):

[0082] f r (h,t)=-||h+rt|| p (1)

[0083] Where p = 1 or p = 2, h, t and r represent the embedding of the head entity, tail entity and the relationship between them, respectively; r represents the transformation vector from h to t; when the triple (h, r, t) is true, the relationship described by the triple in the autonomous driving scenario can be captured from the original knowledge graph through the above scoring function.

[0084] In one implementation, the second text feature may correspond to graph data; wherein, the graph data may include descriptive data of the relationships or object states between at least one scene object contained in the knowledge graph, and the second text feature may be denoted as KG Embedding.

[0085] In one implementation, the second text feature can be obtained in the following way:

[0086] Based on at least one scene object, the objects in the knowledge graph are filtered and selected to obtain the first data. Then, the data in the knowledge graph associated with the first data that represent actions, behaviors, positions and postures are determined as the second data. The set of the first data and the second data is determined as the graph data. Then, the graph data is used to extract features to obtain the second text features.

[0087] SA3 integrates the first text features and the second text features to obtain the first language features.

[0088] In one implementation, language features can be extracted from the first text features and the second text features to obtain the first language features.

[0089] Figure 3B This is a schematic diagram illustrating the process of obtaining first language features as provided in an embodiment of this application. Figure 3B As shown, the process of obtaining the first language feature may include: inputting "pedestrians crossing the zebra crossing with many cars" into the text feature extraction module to obtain the first text feature; simultaneously, processing the above text through a knowledge graph to obtain the text processing result, and then inputting the text processing result into the text feature extraction module to obtain the second text feature; wherein, the text vector may include the first text feature and the second text feature; then, concatenating the first text feature and the second text feature to obtain the text feature concatenation result, and then inputting the text feature concatenation result into a language encoder including LLM and LoRA to obtain the first language feature.

[0090] For example, the first text feature and the second text feature can be concatenated according to the order in which the data appears in the text processing result and the order in which the data appears in the retrieved text.

[0091] Through the above operations, a knowledge graph can be embedded into the search text. Since the data carried in the knowledge graph can include autonomous driving knowledge and scenario description data, each object node in the knowledge graph corresponds to a keyword related to autonomous driving. Thus, through the above operations, the data and state related to autonomous driving scenarios are accurately captured, thereby expanding and enriching the first text features carried by the search text.

[0092] As can be seen from the above, the image determination method provided in this application performs feature extraction processing on the retrieved text to obtain first text features. The retrieved text includes descriptive data for multiple dimensions of the target scene. Thus, through the above steps, discrete text features of the target scene contained in the retrieved text can be obtained. Furthermore, second text features corresponding to the target scene are determined based on a knowledge graph, and the knowledge graph is at least associated with the target scene. This improves the correlation between the second text features, the knowledge graph, and the target scene. At the same time, by leveraging the diversity and comprehensiveness of the data carried in the knowledge graph, the diversity and comprehensiveness of the features in the second text features can be improved. On this basis, integrating the first text features and the second text features to obtain the first language features can improve the comprehensiveness and accuracy of the first language features.

[0093] Based on the foregoing embodiments, the image determination method provided in this application, which determines the second text features corresponding to the target scene based on a knowledge graph, can be implemented in the following ways:

[0094] SC1. Analyze the target scene based on the knowledge graph to determine the set of object parameters.

[0095] The object parameter set includes the state parameters of at least one scene object in the target scene.

[0096] In one implementation, the object parameter set may include parameters contained in the knowledge graph that describe state characteristics such as behavior, position, shape, size, and pose associated with at least one scene object.

[0097] In one implementation, the object parameter set can be determined in the following way:

[0098] Based on the object identifier of at least one scene object, objects in the knowledge graph are filtered to obtain object filtering results. Then, the description parameters, including objects, actions, or states associated with the object filtering results in the knowledge graph, are integrated to obtain an object parameter set. The object identifier may include the number and / or name of the scene object.

[0099] SC2. Perform feature extraction processing on the parameters in the object parameter set to obtain the second text features.

[0100] In one implementation, a feature extraction module can be used to extract features from the parameters in the object parameter set to obtain second text features.

[0101] As can be seen from the above, the image determination method provided in this application analyzes the target scene based on a knowledge graph to determine the object parameter set. The object parameter set includes the state parameters of at least one scene object in the target scene. Thus, the comprehensiveness of the parameters in the object parameter set can be improved by the above method. Furthermore, feature extraction processing is performed on the parameters in the object parameter set to obtain the second text features, which can improve the comprehensiveness of the state features of the scene objects in the second text features.

[0102] Based on the foregoing embodiments, the image determination method provided in this application, which determines the matching result between the first language features and the bird's-eye view feature set corresponding to the image set, can be achieved through the following steps:

[0103] SE1. Perform feature extraction processing on the images in the image set to obtain the first bird's-eye view features.

[0104] The bird's-eye view feature set includes the first bird's-eye view feature.

[0105] In one implementation, the first bird's-eye view feature can be obtained in any of the following ways:

[0106] If the panoramic top-down image is a BEV image in a complex traffic environment associated with an autonomous driving scenario, the panoramic top-down image is encoded by a BEV encoder to obtain the first bird's-eye view feature. In this case, the first bird's-eye view feature may include the BEV feature.

[0107] If the panoramic top-down image is a bird's-eye view image rather than a BEV image, then the first bird's-eye view feature can be obtained by pre-processing the bird's-eye view image through the feature extraction module.

[0108] SE2, Align the first bird's-eye view features and the first language features.

[0109] In one implementation, alignment can be achieved in any of the following ways:

[0110] Align features in the first bird's-eye view feature with features in the first language feature in the time and / or spatial dimensions.

[0111] In the scene object dimension or feature space dimension, the features in the first bird's-eye view feature and the features in the first language feature are aligned so that the m-th feature of the i-th scene object in the first bird's-eye view feature can be aligned with the m-th feature of the i-th scene function object in the first language feature; where i and m are both integers greater than or equal to 1.

[0112] Since the first bird's-eye view feature and the first language feature are located in different feature spaces, the first bird's-eye view feature and the first language feature can be bridged by a set of shared learnable embeddings with shared cross-modal embedding capabilities to weaken the difference between the two features.

[0113] SE3. Match the aligned first bird's-eye view feature with the first language feature to obtain the matching result.

[0114] In one implementation, the matching result can be reflected by the similarity between the aligned first bird's-eye view feature and the first language feature; for example, the similarity can include cosine similarity; for example, at least a portion of the panoramic top-down images in the image set can be filtered based on cosine similarity, and at least one panoramic top-down image with a cosine similarity greater than or equal to a similarity threshold can be identified as the target image.

[0115] As can be seen from the above, the image determination method provided in this application embodiment obtains a first bird's-eye view feature by performing feature extraction processing on the images in the image set, and then aligns the first bird's-eye view feature and the first language feature. In this way, the dimensional difference between the first bird's-eye view feature and the first language feature, which are in different feature spaces, can be reduced, thereby providing data support for determining the matching result between the two and improving the accuracy of the matching result between the two.

[0116] Based on the foregoing embodiments, the image determination method provided in this application, which determines the matching result between the first language features and the bird's-eye view feature set corresponding to the image set, can be implemented in the following ways:

[0117] The matching results are determined by processing the first language features and the bird's-eye view feature set by at least some modules in the retrieval model.

[0118] The retrieval model includes a first extraction module, a second extraction module, a feature alignment module, and a feature matching module. The first extraction module processes the retrieved text to obtain first language features. The second extraction module extracts features from the images in the image set to obtain first bird's-eye view features. The feature alignment module aligns the first bird's-eye view features with the first language features. The feature matching module matches the aligned first bird's-eye view features with the first language features to obtain a matching result. The bird's-eye view feature set includes the first bird's-eye view features.

[0119] Figure 3C This is a schematic diagram of the structure of the retrieval model provided in the embodiments of this application, such as... Figure 3CAs shown, the retrieval model 303 may include two parts: a feature extraction sub-model and a feature alignment sub-model. The feature extraction sub-model may include a BEV Encoder and a Language Encoder associated with LoRA. The BEV Encoder is used to process multiple two-dimensional images to obtain BEV features, i.e., the first bird's-eye view features. The Language Encoder associated with LoRA is used to obtain the first language features through the method provided in the aforementioned embodiment. The Language Encoder may be an LLM.

[0120] For example, the feature alignment module can be a shared cross-modal embedding (SCE), which is used to perform feature alignment processing on the first vector matrix corresponding to the first bird's-eye view feature and the second vector matrix corresponding to the first language feature to obtain the aligned first vector matrix and the second vector matrix.

[0121] For example, the panoramic top-down images in the image set can be filtered by the cosine similarity between the aligned first vector matrix and the second vector matrix to obtain the target image.

[0122] In one implementation, since the retrieval model includes processing of BEV images and retrieved text, the retrieval model can be called a BEV-Text-Scene Retrieval (TSR) model.

[0123] As can be seen from the above, the image determination method provided in this application's embodiments processes the first language features and the bird's-eye view feature set through at least some modules in the retrieval model to determine the matching result. The retrieval model includes a first extraction module, a second extraction module, a feature alignment module, and a feature matching module. Specifically, the first extraction module processes the retrieved text to obtain the first language features; the second extraction module extracts features from the images in the image set to obtain the first bird's-eye view features; the feature alignment module aligns the first bird's-eye view features and the first language features; and the feature matching module matches the aligned first bird's-eye view features and the first language features. Thus, through these modules, integrated processing of the retrieved text and the image set can be achieved. The processing operations of the feature alignment module and the feature matching module can improve the accuracy of the target image.

[0124] Based on the foregoing embodiments, this application also provides a model training method. Figure 4 This is a flowchart illustrating the model training method provided in the embodiments of this application, as shown below. Figure 4 As shown, the method may include the following steps:

[0125] S401. Train the initial retrieval model based on the sample data to obtain the retrieval model.

[0126] The retrieval model is used to determine the matching results between the first language features and the bird's-eye view feature set corresponding to the image set; the first language features are extracted from the retrieval text; and the matching results are used to determine the target image from the image set.

[0127] In one implementation, the sample data includes sample images and sample descriptions; the sample images include panoramic top-down images in various scenarios; the sample descriptions include descriptive data of global features in three-dimensional space for the panoramic top-down images in the sample images in multiple dimensions; the degree of correlation between the panoramic top-down images in the sample images and the descriptive data in the sample descriptions is greater than a degree threshold.

[0128] In one implementation, the panoramic overhead view in the sample image may include a panoramic overhead view of a complex traffic environment in an autonomous or non-autonomous driving scenario; for example, the panoramic overhead view may be a BEV image and / or a bird's-eye view image.

[0129] In one implementation, the sample description may include data that describes the relationships between scene objects in various scenarios, which are context-dependent from both temporal and spatial dimensions. For example, the scene objects may include objects such as vehicles, pedestrians, roads, zebra crossings, traffic lights, overpasses, and traffic signs, and this application embodiment does not limit this.

[0130] In one implementation, the descriptive data in the sample description can comprehensively and multi-dimensionally describe the relationships between scene objects in various scenarios.

[0131] In one implementation, the correspondence between the panoramic top-view image in the sample image and the descriptive data in the sample description can be preset.

[0132] In one implementation, the retrieval model can be obtained in the following way:

[0133] While keeping the parameters of the first extraction module, the second extraction module, and the feature matching module unchanged, the graph text corresponding to the p-th descriptive data in the sample data is determined by combining the knowledge graph. Then, the first extraction module performs feature extraction on the graph text and the p-th descriptive data in the sample data to obtain the initial linguistic features of the p-th descriptive data. The second extraction module performs feature extraction on the panoramic top-down image in the sample data to obtain the initial image features of the sample data. The initial feature alignment module aligns the initial linguistic features of the p-th descriptive data with the initial image features of the sample data to obtain the initial alignment result. Then, the feature matching module determines the degree of matching between the initial image features and the initial linguistic features of the sample data, and the panoramic top-down image in the sample data corresponding to at least two initial image features with the highest matching degree is determined as the initial retrieval result. Then, based on the initial retrieval result and the p-th panoramic top-down image... Based on the differences in the state, the parameters of the initial feature alignment module are adjusted to obtain the first adjusted feature alignment module. The first extraction module, the second extraction module, the first adjusted feature alignment module, and the feature matching module are then determined as the first adjusted retrieval model. Next, based on a method similar to the above process, the parameters of the first adjusted feature alignment module are adjusted until the matching degree between the retrieval result determined by the retrieval model in the training state and the p-th panoramic top-down image is greater than or equal to a preset threshold. At this point, the first extraction module, the second extraction module, the m-th adjusted feature alignment module, and the feature matching module can be determined as the retrieval model. Here, p can be an integer greater than or equal to 1; the p-th panoramic top-down image can include the panoramic top-down image corresponding to the p-th description data in the sample data, and the meaning of the graph text can include the set of object parameters corresponding to the p-th description data in the knowledge graph.

[0134] As can be seen from the above, in the image determination method provided in this application embodiment, an initial retrieval model is trained based on sample data to obtain a retrieval model. Furthermore, the retrieval model is used to determine the matching result between the first language features and the bird's-eye view feature set corresponding to the image set. The first language features are extracted from the retrieval text, and the matching result is used to determine the target image from the image set. This improves the efficiency of determining the matching result and also enhances the flexibility and portability of the process of ultimately determining the target image through the retrieval text and the image set.

[0135] Based on the foregoing embodiments, the model training method provided in this application includes an initial retrieval model comprising a first extraction module, a second extraction module, an initial feature alignment module, and a feature matching module.

[0136] Accordingly, the initial retrieval model is trained based on the sample data to obtain the retrieval model, which can be achieved through the following steps:

[0137] SG1. The first extraction module extracts features from the sample description to obtain the second language features; the second extraction module extracts features from the sample image to obtain the second bird's-eye view features.

[0138] The sample data includes sample images and sample descriptions; the sample images include image data from various scenarios; the sample descriptions include descriptive data of the bird's-eye view features of the image data in the sample images; and the correlation between the image data in the sample images and the descriptive data in the sample descriptions is greater than a threshold.

[0139] In one implementation, the image data in the sample image can be a panoramic top-down view image, and the degree of correlation between the panoramic top-down view image and the descriptive data in the sample description is greater than a degree threshold. This can include a strict and explicit correspondence between the panoramic top-down view image in the sample image and the descriptive data in the sample description. For example, the above correspondence can include a one-to-one relationship or a one-to-many relationship.

[0140] In one implementation, the second language feature can be obtained by extracting features from the sample description and the set of object parameters corresponding to the sample description through the first extraction module during any training process of the initial retrieval model; correspondingly, the second bird's-eye view feature can be obtained by extracting features from the panoramic top-down view image in the sample data through the second extraction module during any training process of the initial retrieval model.

[0141] SG2. The second language features and the second bird's-eye view features are aligned using the initial feature alignment module to obtain the aligned second bird's-eye view features and second language features.

[0142] SG3. The matching module determines the degree of matching between the aligned second bird's-eye view feature and the second language feature; and the sample images are filtered based on the degree of matching to obtain the sample filtering results.

[0143] In one implementation, the degree of matching can correspond to the matching results in the foregoing embodiments, which may include multiple cosine similarities.

[0144] In one implementation, the sample selection result may include a panoramic top-down image obtained by selecting panoramic top-down images from the sample images.

[0145] In one implementation, for the initial feature alignment module, if its input language features can be represented as C = {c1, c2, ..., c...} k The input is the BEV feature, i.e., the second bird's-eye view feature, which is B = {b1, b2, ..., b}. n}, correspondingly, the c in the input language featuresi b in BEV characteristics j The degree of matching between them can be measured by cosine similarity s ij =sim(c i ,b j ) means that, for c i In other words, the maximum cosine similarity can be obtained through r i =max j (s ij The set of cosine similarities can be represented as R = {r1, r2, ..., r...}. k At this point, processing it using the softmax function yields the weights. Therefore, the aligned BEV features can be obtained through the initial feature alignment module. Accordingly, if T represents the second language feature, then the second language feature after feature alignment can be: Where k and n can both be integers greater than 2.

[0146] In this case, the loss of the initial feature alignment module for aligning the second language features and the second bird's-eye view features can be expressed by equations (2) to (4):

[0147]

[0148] Equation (3) is used to characterize the contrast loss between the second language feature and the second bird's-eye view feature, and Equation (4) is used to characterize the contrast loss between the second bird's-eye view feature and the second language feature.

[0149] SG4. The sample screening results are reverse-processed through the description generation module to generate a verification description.

[0150] In one implementation, the description generation module can be used to perform image recognition and text generation processing on the sample screening results to generate a verification description; exemplarily, the description generation (CG) module may include Figure 3C Caption Generation Head in the middle.

[0151] In one implementation, the verification description may include descriptive text that provides a comprehensive description of the global features carried by the sample screening results.

[0152] For example, the loss of the generation module can be expressed by equation (5):

[0153] L CG =CrossEntropy(P logits ,T) (5)

[0154] Among them, P logits It is the logits of the verification description, and T is the second language feature.

[0155] SG5. Based on the difference between the verification description and the sample description, adjust the parameters of the initial feature alignment module to obtain the retrieval model.

[0156] In one implementation, the aforementioned difference state can be characterized by equation (6):

[0157] L = L SCE +λL CG (6)

[0158] Here, λ is used to characterize the weight balance coefficient.

[0159] As can be seen from the above, in the image determination method provided in this application embodiment, the first feature extraction module extracts features from the sample description to obtain second language features, the second extraction module extracts features from the sample image to obtain image features, and the initial feature alignment module aligns the second language features and the second bird's-eye view features to obtain aligned second bird's-eye view features and second language features. Then, the feature matching module determines the matching degree between the aligned second bird's-eye view features and the second language features, and filters the sample images based on the matching degree to obtain sample filtering results. Then, the description generation module performs inverse processing on the sample filtering results to generate verification data. Finally, based on the difference between the verification description and the sample description, the parameters of the initial feature alignment module are adjusted to obtain the retrieval model. The sample data includes sample images and sample descriptions. The sample images include image data from various scenarios, and the sample descriptions include descriptive data of the bird's-eye view features of the image data in the sample images. The correlation between the image data in the sample images and the descriptive data in the sample descriptions is greater than a certain threshold. Thus, by expanding the breadth of sample images included in the sample data, the generalization ability of the retrieval model can be enhanced. By increasing the correlation between image data and sample descriptions beyond a certain threshold, the accuracy of the retrieval model in tracking second language features and second bird's-eye view features can be improved. Furthermore, by introducing a sample generation module to generate verification descriptions, a closed loop between verification descriptions and sample descriptions is achieved. This enables closed-loop adjustment and training of the parameters of the initial feature alignment module in the retrieval model, improving the alignment ability of the feature alignment module for language features and image features. It also enhances the scene generalization ability and common sense reasoning ability of the retrieval model, enabling it to retrieve panoramic bird's-eye view images in autonomous driving scenarios more efficiently and accurately.

[0160] Based on the foregoing embodiments, in the image determination method provided in this application, before training the initial retrieval model based on sample data to obtain the retrieval model, the following steps may also be performed:

[0161] SI1, Obtain the initial sample.

[0162] The initial samples include initial images and initial text; the initial images include two-dimensional images; the initial text includes descriptive data of discrete features of the initial images; and the correlation between the initial images and the initial text is less than a threshold.

[0163] In one implementation, the initial samples may include the nuScenes dataset. The initial text in nuScenes lacks detailed scene information, and while the nuScenes dataset contains over 30,000 samples, it contains only 848 distinct text sentences, resulting in significant duplication among the text descriptions contained therein.

[0164] In one implementation, discrete features may include fragmented, localized descriptions of some features carried by the initial image.

[0165] In one implementation, the degree of association between the initial image and the initial text is less than a degree threshold, which may include the initial image and the initial text not having an explicit correspondence, or the existence of a repetitive correspondence between them.

[0166] SI2: Identify the scene features carried by the initial image to obtain additional text.

[0167] In one implementation, scene features may include features carried in the initial image that correspond to the scene corresponding to the initial image; for example, if the scene corresponding to the initial image is a traffic scene, the scene features may include traffic scene features, such as traffic congestion features, highway traffic conditions, etc.

[0168] In one implementation, the initial image can be extracted and recognized by an image feature extraction module to obtain scene features.

[0169] In one implementation, the additional text may include descriptive text specific to the scene's characteristics.

[0170] In one implementation, the additional text can be obtained in the following way:

[0171] The features in the scene features are subjected to context fusion processing to obtain the feature fusion result, and then additional text is generated based on the feature fusion result.

[0172] SI3. Determine the sample description based on the additional text and the initial text.

[0173] In one implementation, the sample description may include a comprehensive, fine-grained description of the global features carried by the initial image.

[0174] In one implementation, the additional text and the initial text can be context-integrated to obtain a sample description; for example, the sample description can be obtained by extending and augmenting the initial text based on the additional text.

[0175] Figure 5A A comparative illustration of the different levels of sample descriptions provided in the embodiments of this application, such as... Figure 5A As shown, the initial samples can correspond to Original nuScenes, and their data volume can be the first data volume, and the scene features they carry are relatively dispersed.

[0176] For example, by extracting obstacle information from the initial image, quantifying the frequency and number of times obstacles appear, a first supplementary text is obtained. Then, after expanding and augmenting the first supplementary text and the initial text, a first sample description at the nuScenes-Retrieval Easy level can be obtained. The data volume of the first sample description can be a second data volume greater than the first data volume, and the dispersion of the scene features it carries is reduced. For example, the first supplementary text can be "many cars, several trucks, one bus". Correspondingly, the sample data corresponding to the first sample description and its corresponding panoramic overhead image can be denoted as the first sample set.

[0177] For example, by further feature recognition and extraction of the initial image through methods such as object detection, scene classification, self-vehicle decision-making, and decision reasoning, a second supplementary text can be obtained. Then, based on the second supplementary text, the first sample description at the nuScenes-Retrieval Easy level is further expanded and augmented to obtain the second sample description at the nuScenes-Retrieval Hard level. The data volume of the second sample description can be a third data volume greater than the second data volume, and the dispersion of the scene features it carries is further reduced; wherein, the first data volume can be less than 5000, the second data volume can be around 10000, and the third data volume can be around 30000. Correspondingly, the sample data corresponding to the second sample description and its corresponding panoramic top-view image can be denoted as the second sample set.

[0178] SI4. Perform global feature integration on the initial image to obtain the sample image.

[0179] In one implementation, the sample image can be obtained in the following way:

[0180] The image features of the initial image are encoded using a BEV encoder to obtain a sample image representing the panoramic top-down view.

[0181] SI5: Associate the sample description with the sample image to obtain the sample data.

[0182] Among them, the degree of correlation between the image data in the sample image and the descriptive data in the sample description is greater than the degree threshold.

[0183]

[0184] Table 1

[0185] It should be noted that when the sample data contains sample descriptions that are the first sample description and the second sample description, the retrieval model trained on different sample data may have different retrieval performance on the image set based on the retrieval text.

[0186] Table 1 shows the statistical results of the retrieval capabilities of the retrieval model trained based on the first sample set.

[0187] Table 2 shows the statistical results of the retrieval capabilities of the retrieval model trained based on the second sample set.

[0188]

[0189] Table 2

[0190] Tables 1 and 2 compare the retrieval results of BEV-TSR based on BEV space with other image determination methods, including CLIP-ViT-Base, SigLIP-Base, and EVA02-Base, based on six surrounding images (front view and surrounding view). Tables 1 and 2 show that, compared to other image determination methods, BEV-TSR trained on different sample data demonstrates improved image retrieval performance (top-1 accuracy of 85.78% and 87.66%, respectively). While the performance of BEV-TSR trained on the nuScenes-Retrieval Easy dataset decreases slightly compared to the nuScenes-Retrieval Hard dataset, it still outperforms other image determination methods in most metrics. These data indicate that BEV-TSR is superior to other methods shown in the table in handling complex scenes and understanding complex text queries, and it can accurately understand the contextual information contained in the text, thus enabling it to retrieve BEV images in complex traffic scenarios.

[0191] It should be noted that the technical solution provided in this application embodiment can not only retrieve BEV images based on the search text, but also retrieve two-dimensional images based on the search text. Figure 5B This is a schematic diagram illustrating the comparison effect of retrieving two-dimensional images provided in an embodiment of this application, such as... Figure 5B As shown, when the input text query is the same, the BEV-TSR provided in this application embodiment can... Figure 5B The middle image shows the first set of multiple traffic scene images obtained from the retrieval, corresponding to the Text Query. The image set obtained by EVA02 based on the aforementioned text retrieval is... Figure 5B The second image set is shown on the right. From the first and second image sets, it can be seen that the pixel features in the first image set match the scene represented by the Text Query, and the pixel features are context-dependent in the spatial dimension. In contrast, the features carried by the second image set have higher dispersion and poorer context-dependent relationships.

[0192] As can be seen from the above, in the model training method provided in this application embodiment, after obtaining initial samples containing initial images and initial text, wherein the initial text includes discrete features of the initial images, the correlation between the initial images and the initial text is less than a threshold, and the initial images include two-dimensional images, the scene features and object features carried by the initial images are identified, and additional text is obtained. Based on the additional text and the initial text, a sample description is obtained. Thus, through the above operations, the initial text in the initial samples is expanded and broadened in terms of scene features and object features, thereby improving the comprehensiveness and accuracy of the data in the sample description. Furthermore, global feature integration is performed on the initial images to obtain sample images. Thus, through the above operations, the comprehensiveness and diversity of features in the sample images can be improved. On this basis, the sample description and sample images are associated to obtain sample data, and the correlation between the image data in the sample images and the descriptive data in the sample description is greater than a threshold, thereby enhancing the correlation between the sample description and the sample images in the sample data.

[0193] To evaluate the role of each module in the retrieval model during image retrieval, embodiments of this application also provide module ablation results for the retrieval model.

[0194] In this embodiment, the feature alignment module enables SCE to achieve a significant improvement of 5.73% in scene retrieval by aligning BEV and text modalities within a unified embedding space. For example, if a multilayer perceptron (MLP) layer is used to map image features and text features, BEV image retrieval for complex autonomous driving scenarios can still be achieved while keeping the knowledge graph and CG module unchanged.

[0195] In this embodiment of the application, for the first extraction module, in addition to the combination of LoRA and fine-tuned Llama2 in the aforementioned embodiments, a bidirectional transformer model (BERT) can be used instead of the combination of LoRA and fine-tuned Llama2; wherein, Llama2 can be applied to pre-training tasks containing fewer traffic scenarios, and fine-tuning makes the model adapt to the specific context of autonomous driving scenarios.

[0196] In this embodiment, the second extraction module may include a BEV encoder. For example, a BEVDe t or a BEV Former can be used instead of a BEV encoder. Experiments have shown that all three encoders can accurately convert the features of a two-dimensional image to the BEV space.

[0197] Table 3 presents the first statistical results of ablation comparisons of different module combinations provided in the embodiments of this application.

[0198]

[0199] Table 3

[0200] Table 4 presents the second statistical results of ablation comparisons of different module combinations provided in the embodiments of this application.

[0201]

[0202] Table 4

[0203] The specific explanation is as follows:

[0204] BEVFormer outperforms BEVDet in both text and scene retrieval tasks, demonstrating its superior effectiveness in transforming 2D features into 3D space. The Llama2 model fine-tuned with LoRA outperforms BERT and the untuned Llama2 model in both text and scene retrieval tasks, highlighting the importance of fine-tuning for improving model performance. DistMult performs best in knowledge graph embedding and is therefore selected as the default knowledge graph embedding extractor. For SCE, model performance improves with increasing embedding vector dimension, achieving optimal performance at a vector dimension of 4096; thus, a larger embedding space helps to better align features from different modalities.

[0205] In summary, the retrieval model provided in this application is a novel BEV-TSR framework, which can perform BEV image retrieval on BEV images received from retrieval text in autonomous driving scenarios. During the retrieval process, it can fully understand the global context of the retrieval text, enabling it to retrieve BEVs corresponding to complex traffic scenarios. In the process of processing the retrieval text, the combination of LLM and knowledge graphs achieves a comprehensive, accurate, and in-depth understanding of the retrieval text, thereby enabling the first language features to possess high-level, rich semantics. Furthermore, by introducing learnable embeddings shared in shared cross-modal embeddings to bridge the gap between BEV features and language embeddings, the accuracy of subsequent feature matching is improved.

[0206] Furthermore, during the training process for BEV-TSR, the alignment can be further enhanced by combining it with the CG model. At the same time, during the training process, the multi-level retrieval dataset nuScenes-Retrieval, built on the nuScenes dataset, enables BEV-TSR to achieve more accurate and efficient BEV image retrieval.

[0207] Based on the foregoing embodiments, this application also provides an image determination device. Figure 6 This is a schematic diagram of the structure of the image determination device provided in the embodiments of this application, such as... Figure 6 As shown, the image determining device 600 includes:

[0208] Processing module 601 is used to extract first language features from the retrieved text;

[0209] The determination module 602 is used to determine the matching result between the first language features and the bird's-eye view feature set corresponding to the image set; based on the matching result, the target image is determined from the image set.

[0210] In some embodiments, the processing module 601 is used to perform feature extraction processing on the retrieved text to obtain first text features;

[0211] The processing module 601 is used to determine the second text feature corresponding to the target scene based on the knowledge graph; integrate the first text feature and the second text feature to obtain the first language feature; wherein the knowledge graph is at least associated with the target scene.

[0212] In some embodiments, the processing module 601 is used to analyze the target scene based on the knowledge graph to determine the object parameter set; and to perform feature extraction processing on the parameters in the object parameter set to obtain a second text feature; wherein the object parameter set includes the state parameters of at least one scene object in the target scene.

[0213] In some embodiments, the processing module 601 is used to perform feature extraction processing on the images in the image set to obtain a first bird's-eye view feature; to perform alignment processing on the first bird's-eye view feature and the first language feature; to match the aligned first bird's-eye view feature and the first language feature to obtain a matching result; the bird's-eye view feature set includes the first bird's-eye view feature.

[0214] In some embodiments, the determining module 602 is used to process the first language features and the bird's-eye view feature set through at least some modules in the retrieval model to determine the matching result; wherein, the retrieval model includes a first extraction module, a second extraction module, a feature alignment module, and a feature matching module; the first extraction module is used to process the retrieved text to obtain the first language features; the second extraction module is used to extract features from the images in the image set to obtain the first bird's-eye view features; the feature alignment module is used to align the first bird's-eye view features and the first language features; the feature matching module is used to match the aligned first bird's-eye view features and the first language features to obtain the matching result.

[0215] Based on the foregoing embodiments, this application also provides a model training apparatus. Figure 7 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application, as shown below. Figure 7 As shown, the model training device 700 may include:

[0216] The training module 701 is used to train the initial retrieval model based on sample data to obtain the retrieval model; wherein, the retrieval model is used to determine the matching result between the first language features and the bird's-eye view feature set corresponding to the image set; the first language features are used to extract from the retrieval text; the matching result is used to determine the target image from the image set.

[0217] In some embodiments, the initial retrieval model includes a first extraction module, a second extraction module, an initial feature alignment module, and a feature matching module;

[0218] Training module 701 is used to extract features from sample descriptions through a first extraction module to obtain second language features; extract features from sample images through a second extraction module to obtain second bird's-eye view features; align the second language features and second bird's-eye view features through an initial feature alignment module to obtain aligned second bird's-eye view features and second language features; determine the matching degree between the aligned second bird's-eye view features and second language features through a feature matching module, and filter sample images based on the matching degree to obtain sample filtering results; perform inverse processing on the sample filtering results through a description generation module to generate a verification description; adjust the parameters of the initial feature alignment module based on the difference between the verification description and the sample description to obtain a retrieval model; wherein, sample data includes sample images and sample descriptions; sample images include image data from various scenarios; sample descriptions include descriptive data of bird's-eye view features of images in sample images; and the correlation between images in sample images and descriptive data in sample descriptions is greater than a threshold.

[0219] In some embodiments, the training module 701 is used to obtain initial samples; wherein the initial samples include an initial image and initial text; the initial image includes a two-dimensional image; the initial text includes descriptive data of discrete features of the initial image; and the degree of correlation between the initial image and the initial text in the initial samples is less than a degree threshold.

[0220] The training module 701 is also used to identify the scene features carried by the initial image to obtain additional text; determine the sample description based on the additional text and the initial text; perform global feature integration on the initial image to obtain the sample image; associate the sample description with the sample image to obtain sample data; wherein the degree of association between the image data in the sample image and the descriptive data in the sample description is greater than the degree threshold.

[0221] Based on the foregoing embodiments, this application also provides an electronic device. Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802; the memory 802 stores a computer program; when the computer program is executed by the processor 801, it can implement any of the aforementioned image determination methods or model training methods.

[0222] Based on the foregoing embodiments, this application also provides a computer-readable storage medium storing a computer program; when the computer program is executed by the processor of an electronic device, it can implement any of the aforementioned image determination methods or model training methods.

[0223] Based on the foregoing embodiments, this application also provides a computer program product, which includes a computer program; when the computer program is executed by the processor of an electronic device, it can implement any of the aforementioned image determination methods or model training methods.

[0224] In some embodiments, the computer-readable storage medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it may be a device that includes one or any combination of the above-mentioned memories.

[0225] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0226] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

Claims

1. An image determination method, characterized in that, The method includes: Extracting first language features from the retrieved text; Determine the matching result between the first language features and the bird's-eye view feature set corresponding to the image set; Based on the matching results, the target image is determined from the image set.

2. The method according to claim 1, characterized in that, The extraction of first language features from the retrieved text includes: The search text is subjected to feature extraction processing to obtain the first text feature; wherein, the search text includes descriptive data of multiple dimensions for the target scene; A second text feature corresponding to the target scene is determined based on a knowledge graph; wherein the knowledge graph is at least associated with the target scene; The first text feature and the second text feature are integrated to obtain the first language feature.

3. The method according to claim 2, characterized in that, The determination of the second text features corresponding to the target scene based on the knowledge graph includes: The target scene is analyzed based on the knowledge graph to determine the object parameter set; wherein, the object parameter set includes the state parameters of at least one scene object in the target scene; The parameters in the object parameter set are subjected to feature extraction processing to obtain the second text feature.

4. The method according to any one of claims 1-3, characterized in that, The step of determining the matching result between the first language features and the bird's-eye view feature set corresponding to the image set includes: Feature extraction processing is performed on the images in the image set to obtain a first bird's-eye view feature; the bird's-eye view feature set includes the first bird's-eye view feature; Alignment processing is performed on the first bird's-eye view feature and the first language feature; The first bird's-eye view feature and the first language feature are matched and aligned to obtain the matching result.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the matching result between the first language features and the bird's-eye view feature set corresponding to the image set includes: The matching result is determined by processing the first language features and the bird's-eye view feature set using at least some modules in the retrieval model; wherein, the retrieval model includes a first extraction module, a second extraction module, a feature alignment module, and a feature matching module; the first extraction module is used to process the retrieved text to obtain the first language features; the second extraction module is used to extract features from the images in the image set to obtain the first bird's-eye view features; the feature alignment module is used to align the first bird's-eye view features and the first language features; the feature matching module is used to match the aligned first bird's-eye view features and the first language features to obtain the matching result; the bird's-eye view feature set includes the first bird's-eye view features.

6. A model training method, characterized in that, The method includes: The initial retrieval model is trained based on sample data to obtain a retrieval model; wherein, the retrieval model is used to determine the matching result between the first language features and the bird's-eye view feature set corresponding to the image set; the first language features are extracted from the retrieval text; the matching result is used to determine the target image from the image set.

7. The method according to claim 6, characterized in that, The initial retrieval model includes a first extraction module, a second extraction module, an initial feature alignment module, and a feature matching module; The process of training the initial retrieval model based on sample data to obtain the retrieval model includes: The first extraction module extracts features from the sample description to obtain second language features; wherein, the sample data includes sample images and sample descriptions; the sample images include image data from various scenarios; the sample descriptions include descriptive data of bird's-eye view features of the image data in the sample images; the correlation between the image data in the sample images and the descriptive data in the sample descriptions is greater than a degree threshold; The second extraction module extracts features from the sample image to obtain the second bird's-eye view feature; The second language feature and the second bird's-eye view feature are aligned by the initial feature alignment module to obtain the aligned second bird's-eye view feature and the second language feature. The feature matching module determines the degree of matching between the aligned second bird's-eye view feature and the second language feature, and filters the sample images based on the degree of matching to obtain the sample filtering results; The sample screening results are reverse-processed by the description generation module to generate a verification description. Based on the difference between the verification description and the sample description, the parameters of the initial feature alignment module are adjusted to obtain the retrieval model.

8. The method according to claim 6 or 7, characterized in that, Before training the initial retrieval model based on sample data to obtain the retrieval model, the method further includes: Obtain initial samples; wherein, the initial samples include an initial image and initial text; the initial image includes a two-dimensional image; the initial text includes descriptive data of discrete features of the initial image; the correlation between the initial image and the initial text is less than a degree threshold; Identify the scene features carried by the initial image to obtain additional text; The sample description is determined based on the additional text and the initial text; Global feature integration is performed on the initial image to obtain a sample image; The sample data is obtained by associating the sample description with the sample image; wherein the degree of correlation between the image data in the sample image and the descriptive data in the sample description is greater than a degree threshold.

9. An image determining device, characterized in that, The image determination device includes: The processing module is used to extract first language features from the retrieved text; The determining module is used to determine the matching result between the first language features and the bird's-eye view feature set corresponding to the image set; and to determine the target image from the image set based on the matching result.

10. A model training device, characterized in that, The model training device includes: The training module is used to train an initial retrieval model based on sample data to obtain a retrieval model; wherein, the retrieval model is used to determine the matching result between a first language feature and a bird's-eye view feature set corresponding to the image set; the first language feature is extracted from the retrieval text; the matching result is used to determine the target image from the image set.

11. An electronic device, characterized in that, The electronic device includes a processor and a memory; the memory stores a computer program; when the computer program is executed by the processor, it can implement the image determination method as described in any one of claims 1 to 5, or the model training method as described in any one of claims 6 to 8.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program; when the computer program is executed by the processor of the electronic device, it can implement the image determination method as described in any one of claims 1 to 5, or the model training method as described in any one of claims 6 to 8.

13. A computer program product, characterized in that, The program product includes a computer program; when the computer program is executed by the processor of an electronic device, it is capable of implementing the image determination method as described in any one of claims 1 to 5, or the model training method as described in any one of claims 6 to 8.

Images