Data processing method, image retrieval method, electronic device, and storage medium

Abstract

This application discloses an image retrieval method, apparatus, electronic device, and readable storage medium. The method includes: acquiring retrieval text; the retrieval text describing attribute information of multiple entities and the relationship between behaviors to be retrieved; the multiple entities including a target vehicle and related retrieval entities; related retrieval entities including traffic objects related to the driving behavior of the target vehicle; the relationship between behaviors to be retrieved including the relationship between behaviors between different entities; constructing a retrieval knowledge graph based on the retrieval text; the retrieval knowledge graph indicating the relationship between behaviors to be retrieved and the attribute information of multiple entities; and performing image retrieval using the retrieval text and the retrieval knowledge graph as indexes to obtain the target image corresponding to the retrieval text. According to the method of this application, the accuracy of the retrieved target images is high.

Description

Technical Field

[0001] This application relates to the field of electronic information technology, and more specifically, to a data processing method, an image retrieval method, an apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the development of autonomous driving functions, more and more driving images need to be retrieved in order to optimize autonomous driving algorithms based on the retrieved driving images.

[0003] However, existing image retrieval methods have low accuracy in retrieving driving images, resulting in poor optimization performance when using driving images to optimize autonomous driving algorithms. Summary of the Invention

[0004] This application proposes a data processing method, an image retrieval method, an apparatus, an electronic device, and a computer-readable storage medium to improve the retrieval accuracy of driving images.

[0005] In a first aspect, embodiments of this application provide an image retrieval method, the method comprising:

[0006] Obtain the search text; the search text is used to describe the attribute information of multiple entities and the relationship between the searched behaviors; the multiple entities include the target vehicle and related search entities; the related search entities include traffic objects related to the driving behavior of the target vehicle; the relationship between the searched behaviors includes the relationship between behaviors between different entities;

[0007] A retrieval knowledge graph is constructed based on the retrieved text; the retrieval knowledge graph is used to indicate the relationships between the retrieved behaviors and the attribute information of multiple entities;

[0008] Image retrieval is performed using the search text and the search knowledge graph as indexes to obtain the target image corresponding to the search text.

[0009] Secondly, embodiments of this application also provide an image retrieval device, the device comprising:

[0010] The acquisition module is used to acquire the search text; the search text describes the attribute information of multiple entities and the relationship between the searched behaviors; the multiple entities include the target vehicle and related search entities; the related search entities include traffic objects related to the driving behavior of the target vehicle; the relationship between the searched behaviors includes the relationship between behaviors between different entities;

[0011] The knowledge graph determination module is used to construct a retrieval knowledge graph based on the retrieved text; the retrieval knowledge graph is used to indicate the relationships between the retrieved behaviors and the attribute information of multiple entities;

[0012] The retrieval module is used to perform image retrieval using the retrieval text and the retrieval knowledge graph as indexes, and to obtain the target image corresponding to the retrieval text.

[0013] Thirdly, embodiments of this application also provide an electronic device, which includes: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the methods described above.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing processor-executable program code, which, when executed by the processor, causes the processor to perform the above-described method.

[0015] This application provides an image retrieval method, apparatus, electronic device, and computer-readable storage medium. Firstly, a retrieval knowledge graph is constructed based on the acquired retrieval text. This knowledge graph indicates the relationships between the retrieved behavior and the attribute information of multiple entities. Thus, when the graph-structured retrieval knowledge graph and the retrieval text are used together as the image retrieval index, a combined image-text indexing method is achieved. This combined indexing can more accurately and completely indicate the relationships between the retrieved behavior and the attribute information of multiple entities. Therefore, when retrieving images using both the retrieval text and the retrieval knowledge graph, the retrieved target image is more closely matched to the relationships between the retrieved behavior and the attribute information of multiple entities. The target image has a higher degree of fit with the retrieval text, resulting in higher accuracy. Consequently, when optimizing autonomous driving algorithms based on the target image, the optimization effect is better.

[0016] Other features and advantages of the embodiments of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the embodiments of this application. The objects and other advantages of the embodiments of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of an image retrieval method according to an embodiment of this application is shown.

[0019] Figure 2A flowchart of yet another image retrieval method proposed in one embodiment of this application is shown.

[0020] Figure 3 A schematic diagram of a knowledge graph according to an embodiment of this application is shown.

[0021] Figure 4 A schematic diagram of an image retrieval process according to an embodiment of this application is shown.

[0022] Figure 5 A structural block diagram of an image retrieval device according to an embodiment of this application is shown.

[0023] Figure 6 A structural block diagram of an electronic device provided according to an embodiment of this application is shown. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. The components of the embodiments of the present application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive effort are within the scope of protection of the present application.

[0025] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] Please see Figure 1 , Figure 1 A flowchart of an image retrieval method according to an embodiment of this application is shown, for use in an electronic device. The method includes:

[0027] S110. Obtain the search text; construct a search knowledge graph based on the search text.

[0028] The search text describes the attribute information of multiple entities and the relationships between the searched behaviors; the multiple entities include the target vehicle and related search entities; the related search entities include traffic objects related to the driving behavior of the target vehicle; the relationships between the searched behaviors include the behavioral relationships between different entities; and the search knowledge graph is used to indicate the relationships between the searched behaviors and the attribute information of multiple entities.

[0029] In this application, the vehicle can be a fuel-powered vehicle, an electric vehicle, or a hybrid vehicle. According to the vehicle type classification, the vehicle can also be a sedan, a bus, or a truck, etc.

[0030] The target vehicle refers to the subject of the image retrieved during image retrieval. In other words, the retrieved target image is an image taken by the target vehicle.

[0031] Relevant search entities include traffic objects related to the target vehicle's driving behavior, such as vehicles, pedestrians, lane markings (e.g., zebra crossings, stop lines), and traffic lights.

[0032] For any traffic object, if its driving behavior affects the driving behavior of the target vehicle, it is identified as a relevant search entity. For example, the vehicle following the target vehicle can be a relevant search entity, other vehicles in the adjacent lane to the left of the target vehicle can be relevant search entities, and pedestrians in front of the target vehicle to the left can be relevant search entities.

[0033] The attribute information of an entity is used to describe the state of the entity. For entities whose position changes (such as vehicles and pedestrians), the attribute information may include entity type (entity type such as truck, bus, car, pedestrian, etc.), location information, heading angle, acceleration, speed, distance to the lane of the target vehicle (the vehicle that collects driving images and videos), and distance to other vehicles. For entities whose position remains unchanged (such as zebra crossings, navigation lines, and traffic lights), the attribute information may include the entity's location information and entity name.

[0034] Behavioral association refers to the relationship between the behaviors of different objects. For example, the behavioral association between a target vehicle and the vehicle in front in its lane is "following"; the behavioral association between a target vehicle and the vehicle behind in the adjacent lane is "adjacent behind"; and the behavioral association between a pedestrian and a zebra crossing is "the pedestrian is on the zebra crossing." Correspondingly, the behavioral association to be retrieved indicates the behavioral relationship between different entities mentioned in the search text (including the target vehicle and related search entities).

[0035] The search text can be generated based on actual needs, and can be any language form of text that can describe the attribute information of multiple entities and the relationship between the searched behavior. It can be text in English, Korean, or Chinese characters. This application does not limit it.

[0036] A retrieval knowledge graph is a graph structure that indicates the relationships between the retrieved behaviors and the attribute information of multiple entities. A retrieval knowledge graph can include entity nodes and node connecting edges. Entity nodes represent a single entity, and node connecting edges indicate the behavioral relationships between the entities represented by the entity nodes. The node name of an entity node can be the name or number of the entity it represents. In the retrieval knowledge graph, entity attribute information can be added as supplementary information or attributes to the node attributes of the corresponding entity node.

[0037] Therefore, the aforementioned construction of a retrieval knowledge graph based on the retrieved text may include: determining the entity node corresponding to each entity according to the attribute information of each entity; and for any two entities with behavioral association, constructing node connection edges between the two entity nodes corresponding to the two entities according to the behavioral association between the two entities, so as to obtain the retrieval knowledge graph.

[0038] Generally, different node connection edges are determined for different behavioral relationships. Of course, the same connection edge form can be used to indicate the node connection edges corresponding to different behavioral relationships, and the node connection edges can be labeled or have attributes added by the relationship identifier corresponding to the behavioral relationship (such as the name or code of the behavioral relationship) to distinguish different node connection edges.

[0039] S120. Using the search text and the search knowledge graph as indexes, perform image retrieval to obtain the target image corresponding to the search text.

[0040] After obtaining the retrieval knowledge graph, both the retrieval text and the retrieval knowledge graph can be used as indexes for image retrieval, and the retrieved images can be used as target images for the corresponding retrieval text.

[0041] One approach is to use the search text as an index to retrieve multiple first candidate images from a given database, and then use the search knowledge graph as an index to retrieve one of the multiple first candidate images as the target image.

[0042] Here, the similarity between the searched text and each image in the given database can be determined (such as cosine similarity, Euclidean distance, and Manhattan distance). Then, multiple first candidate images are selected according to the similarity from high to low. Alternatively, multiple first candidate images with similarity higher than a first threshold (such as 0.5) can be selected. Then, the similarity between the searched knowledge graph and the multiple first candidate images can be determined (such as cosine similarity, Euclidean distance, and Manhattan distance). The one with the highest similarity is selected as the target image.

[0043] Similarly, the knowledge graph can be used as an index to retrieve multiple second candidate images from a given database, and then the retrieval text can be used as an index to retrieve one of the multiple second candidate images as the target image.

[0044] Of course, here we can determine the similarity between the retrieved knowledge graph and each image in the given database (which can be cosine similarity, Euclidean distance, Manhattan distance, etc.), and then select multiple second candidate images in descending order of similarity. Alternatively, we can select multiple second candidate images with similarity higher than a second threshold (e.g., 0.55), and then determine the similarity between the retrieved text and multiple second candidate images (which can be cosine similarity, Euclidean distance, Manhattan distance, etc.), and select the one with the highest similarity as the target image.

[0045] In this embodiment, a retrieval knowledge graph is first constructed based on the acquired retrieval text. This retrieval knowledge graph indicates the relationship between the retrieved behavior and the attribute information of multiple entities. Thus, when the graph-structured retrieval knowledge graph and the retrieval text are used together as the index for image retrieval, a combined text-image indexing method is achieved. This combined text-image index can more accurately and completely indicate the relationship between the retrieved behavior and the attribute information of multiple entities. Therefore, when retrieving images by combining the retrieval text and the retrieval knowledge graph, the retrieved target image can better match the relationship between the retrieved behavior and the attribute information of multiple entities. The target image has a higher degree of fit with the retrieval text, and the accuracy of the target image is higher. Consequently, when optimizing autonomous driving algorithms based on the target image, the optimization effect is better.

[0046] In some embodiments, the target image is retrieved from a database, which includes multiple driving images and their respective image retrieval indexes, such as... Figure 2 As shown, before S120, the method further includes:

[0047] S210. Obtain the image description information and target entity corresponding to the driving image.

[0048] In this application, the "main vehicle" refers to the vehicle for which driving images are acquired. The target entities include the main vehicle and traffic objects related to the driving behavior of the main vehicle.

[0049] When a traffic object's driving behavior affects the driving behavior of the main vehicle, it is identified as a target entity. For example, the vehicle following the main vehicle can be a target entity, other vehicles in the adjacent lane to the left of the main vehicle can be target entities, and pedestrians in front of the main vehicle to the left can be target entities.

[0050] In other words, the main vehicle in the captured driving image can be regarded as a target entity in the driving image, and traffic objects in the driving image that are related to the driving behavior of the main vehicle can also be regarded as target entities corresponding to the driving image.

[0051] The image description information corresponding to the driving image refers to the descriptive information used to describe the content of the driving image. For example, the image description information corresponding to the driving image could be "Scene of the main vehicle changing lanes to the left: There is a pedestrian crossing the zebra crossing in front of the main vehicle, there is a car to the left front of the main vehicle, and a large truck accelerating to the left rear of the main vehicle."

[0052] In this embodiment, image description information can be obtained by technicians analyzing driving images, or by analyzing driving images using an image description model, such as a clip model.

[0053] In some embodiments, the method for acquiring driving images may include: identifying traffic scenes in the driving video to be processed; if a target traffic scene is identified in the driving video, determining a time window corresponding to the target traffic scene based on the scene behavior video segment corresponding to the target traffic scene; wherein, the scene behavior video segment refers to a video segment in the driving video that includes the target driving behavior; the target driving behavior is the driving behavior performed by the main vehicle in response to the target traffic scene; acquiring the video segment located in the time window corresponding to the target traffic scene from the driving video as the target video segment; and acquiring the video frames in the target video segment as driving images.

[0054] Driving video refers to video footage collected by a vehicle during its journey using its own cameras and other means, capturing images of the vehicle's driving environment. This video can be collected at any point during the vehicle's journey; for example, it could be video collected between 1 PM and 2 PM on December 1, 2021, or it could be video collected while the vehicle is driving on a highway.

[0055] The traffic scenario can be the scenario where the main vehicle is located, and can include turning, straight-going, lane-changing, merging into a lane, etc. The target traffic scenario can be one or more traffic scenarios selected according to requirements.

[0056] Traffic scene recognition can be performed on driving videos using established rules or scene recognition models. Established rules can refer to the determination that there is a target driving behavior in the target traffic scene (for example, the target driving behavior in a turning scene is turning behavior), and the target traffic scene can be identified. The scene recognition model can be a model built on a neural network to identify the target traffic scene. The scene recognition model can be trained on a neural network model with parameters initialized by sample traffic scenes and sample videos belonging to the sample traffic scenes.

[0057] If the driving video identifies a video segment of the target traffic scene, it means that the scene behavior video segment of the target traffic scene has been detected. At this time, it is necessary to extract the target video segment corresponding to the target traffic scene from the driving video.

[0058] As mentioned above, a scene behavior video segment refers to a video segment in a driving video that includes the target driving behavior; the target driving behavior is the driving behavior performed by the main vehicle in response to the target traffic scene. For example, if the target traffic scene is a lane-changing scene, the target driving behavior for the target traffic scene is the lane-changing behavior of the main vehicle; or if the target traffic scene is a turning scene, the target driving behavior for the target traffic scene is the turning behavior of the main vehicle.

[0059] However, when a driver reacts to any action while driving the vehicle, there is a reaction time required. In other words, the driver has already made a decision about the driving behavior before the vehicle takes any action. Therefore, it is necessary to set a time window for the traffic scenario to indicate the start and end time of the driver's decision regarding the traffic scenario.

[0060] In this application, the start time of a time window in a traffic scenario is the time point extending forward by a first duration from the time point when the main vehicle begins its driving behavior in that traffic scenario (i.e., the start time point of the scene behavior video segment), and the end time of the time window in that traffic scenario is the time point extending backward by a second duration from the time point when the main vehicle completes its driving behavior in that traffic scenario (i.e., the end time point of the scene behavior video segment); wherein, the first duration and the second duration are determined based on the driver's reaction time to the traffic object, and the first duration and the second duration can be the same.

[0061] Generally, the first and second time windows can vary depending on the traffic scenario. For example, for a turning scenario, both the first and second time windows are 5 seconds, while for a lane-changing scenario, both the first and second time windows are 3 seconds. Therefore, the length of the time window can vary depending on the traffic scenario.

[0062] After obtaining the time window corresponding to the target traffic scene, the video segment located in the time window corresponding to the target traffic scene in the driving video is directly obtained as the target video segment. Then, the video frames in the target video segment are obtained as driving images. At this time, the driving images are driving images under the target traffic scene.

[0063] In this application, all video frames in the target video segment can be obtained as driving images, or a portion of video frames can be sampled from the target video segment as driving images.

[0064] In some embodiments, the method for obtaining image description information corresponding to a driving image may include: adding text descriptions to the driving image based on target prompt information corresponding to the target traffic scene; and determining image description information corresponding to the driving image based on the text descriptions added to the driving image and the scene description information of the target traffic scene.

[0065] The prompt information is used to provide guidance on the generated text descriptions, specifying the requirements or scope that the generated text descriptions must meet. In other words, target prompt information can refer to prompt information set for a specific traffic scenario, used to indicate the requirements or scope that the text descriptions corresponding to the driving images must meet.

[0066] Typically, different prompts are set for different traffic scenarios. Of course, the same prompts can also be set for different traffic scenarios, so that the text descriptions obtained in various traffic scenarios meet the same requirements or scope.

[0067] A text description model can be used to add text descriptions to driving images based on target prompts corresponding to the target traffic scene. Then, the added text descriptions and the scene description information of the target traffic scene are obtained as the image description information corresponding to the driving image. Here, the text description model can be a clip model, etc., and the scene description information refers to the text information describing the traffic scene.

[0068] In other words, the image description information determined for a driving image can include textual descriptions added to the driving image and scene descriptions of the target traffic scene in which the driving image is located. The traffic scene is actually a temporal expression (for example, if the traffic scene is a turning scene, the scene description information is used to indicate the dynamic process of turning involved in the turning scene), while the textual description is a spatial expression—the various objects in the driving image and the positional relationships between them (for example, there is an obstacle vehicle to the left front of the main vehicle). Thus, a relatively rich description is added to the driving image: that is, the scene description information describes the content of the driving image in the temporal dimension, and the explanatory text describes the content of the driving image in the spatial dimension, making the image description information of the driving image more consistent with the actual driving situation of the main vehicle, and the accuracy of the image description information of the driving image is high.

[0069] In some embodiments, before S210, the method further includes: determining the target traffic scene where the main vehicle is located when the driving image is acquired; determining the driver reaction distance of the main vehicle's driver based on the target traffic scene; and obtaining, from traffic objects related to the driving behavior of the main vehicle, traffic objects whose distance from the main vehicle does not exceed the driver reaction distance as target entities.

[0070] Driver reaction distance refers to the distance the vehicle travels from when the driver sees an event occurring until the driver reacts to it. Generally, the driver's reaction distance varies depending on the traffic scenario. For example, the driver's reaction distance is 20 meters for turning, and 50 meters for changing lanes.

[0071] Therefore, for the target traffic scenario, the driver's reaction distance is determined, and then traffic objects that are within the driver's reaction distance of the vehicle capturing the driving video and are related to the vehicle's driving behavior are identified as target entities. Thus, a target entity is a subset of traffic objects related to the vehicle's driving behavior, or it could be all of them. This means that all traffic objects related to the vehicle's driving behavior are within the driver's reaction distance.

[0072] Generally, multiple driving images are selected from driving videos. For the same target video segment in the same driving video, since the position of the main vehicle in the target video segment is constantly changing, the target entity identified after determining different driving images in the target video segment may be different.

[0073] S220. Based on the target behavior association and the attribute information of the target entity, determine the knowledge graph corresponding to the driving image.

[0074] Among them, the target behavior relationship includes the behavior relationship between different target entities; the knowledge graph is used to indicate the target behavior relationship and the attribute information of the target entities.

[0075] The attribute information of an entity is used to describe the state of the entity. For entities whose position changes (such as vehicles and pedestrians), the attribute information may include entity type (entity type such as truck, bus, car, pedestrian, etc.), location information, heading angle, acceleration, speed, distance from the lane of the main vehicle (the vehicle that collects driving images and videos), and distance from other vehicles. For entities whose position remains unchanged (such as zebra crossings, navigation lines, and traffic lights), the attribute information may include the entity's location information and entity name.

[0076] In this embodiment, the attribute information of the target entity can be obtained from the driving video or determined based on driving data collected by other sensors of the main vehicle (such as radar, gyroscope, wheel speed pulse sensor, etc.). For example, the acceleration of the main vehicle at the time point of the driving image is determined based on the data collected by the acceleration sensor, and the speed and acceleration of other vehicles at the time point of the driving image are determined by the data collected by the radar.

[0077] Of course, the attribute information determined for the same target entity may differ in different traffic scenarios. For example, in a turning scenario, the attribute data of other vehicles may include position information, heading angle, acceleration, speed, turn signal status information, distance from the lane where the main vehicle is located, and distance from other vehicles. Similarly, in a lane changing scenario, the attribute data of other vehicles may include position information, heading angle, acceleration, speed, distance from the lane where the main vehicle is located, and distance from other vehicles.

[0078] After determining the attribute information of each target entity in the driving image, the knowledge graph corresponding to the driving image can be determined based on the behavioral relationships between different target entities and the attribute information of each target entity.

[0079] In this embodiment, the behavioral association between different target entities can be determined based on at least one of the image description information and the driving image, so as to obtain the target behavioral association.

[0080] In other words, based on at least one of image description information and driving images, the behavioral relationships between different target entities are determined, and these determined behavioral relationships are summarized into a general relationship as the target behavioral relationship. For example, if the image description information determines that vehicle cc1 follows the main vehicle, and the driving images determine that vehicle cc2 follows vehicle cc1, the resulting target behavioral relationship could be either vehicle cc1 following the main vehicle, or both vehicle cc1 following the main vehicle and vehicle cc2 following vehicle cc1.

[0081] A knowledge graph can include entity nodes and node connecting edges. Entity nodes indicate a target entity, and node connecting edges indicate the behavioral relationships between the target entities represented by the entity nodes. The node name of an entity node can be the name or number of the target entity it indicates. In a knowledge graph, the attribute information of the target entity can be added to the node attributes of the corresponding entity node as supplementary information or attributes.

[0082] In some implementations, S220 may include: determining the entity node corresponding to each target entity based on the attribute information of each target entity; and for any two target entities with behavioral association, constructing node connection edges between the two entity nodes corresponding to the two target entities based on the behavioral association between the two target entities to obtain the knowledge graph corresponding to the driving image.

[0083] Generally, different node connection edges are determined for different behavioral relationships. Of course, the same connection edge form can be used to indicate the node connection edges corresponding to different behavioral relationships, and the node connection edges can be labeled or have attributes added by the relationship identifier corresponding to the behavioral relationship (such as the name or code of the behavioral relationship) to distinguish different node connection edges.

[0084] Of course, not every two target entities have a behavioral relationship. Therefore, for any two target entities that have a behavioral relationship, a node connection edge is constructed between the two entity nodes corresponding to the two target entities based on the behavioral relationship between the two target entities. For any two target entities that do not have a behavioral relationship, no node connection edge needs to be constructed.

[0085] For example, for a driving image in a turning scene, the target entities involved include the main vehicle, obstacle vehicle 1 (located in front of the main vehicle in the adjacent lane), obstacle vehicle 2 (located behind the main vehicle in the adjacent lane), pedestrian, zebra crossing, and following vehicles (vehicles following the main vehicle). The target behavior relationships include: obstacle vehicle 2 following obstacle vehicle 1, pedestrian in front of the main vehicle, pedestrian on the zebra crossing, following vehicle following the main vehicle, obstacle vehicle 2 adjacent to the main vehicle behind it, and obstacle vehicle 1 adjacent to the main vehicle in front of it. The determined knowledge graph is as follows: Figure 3 As shown, the nodes are labeled with their behavioral relationships. Of course, entity nodes include the attribute information of the corresponding target entity. For clarity and simplicity, in... Figure 3 It is not shown in the text.

[0086] S230. Determine the image retrieval index corresponding to the driving image based on the image description information and knowledge graph.

[0087] After obtaining the image description information and knowledge graph of the driving image, the image description information and knowledge graph of the driving image can be used as the image retrieval index corresponding to the driving image. Then, the image retrieval index corresponding to the driving image can be stored in the database to facilitate subsequent image retrieval or autonomous driving simulation processing based on the image retrieval index corresponding to the driving image.

[0088] Of course, in some embodiments, S230 may include: extracting text features from the image description information to obtain text feature information; extracting graph features from the knowledge graph to obtain graph feature information; and determining the image retrieval index corresponding to the driving image based on the text feature information and the graph feature information.

[0089] In this application, text features can be extracted from image description information using a text representation model (e.g., Chinese-BERT) to obtain text feature information. Then, graph features can be extracted from the knowledge graph using a Large Language Model (LLM) to obtain graph feature information. Finally, the text feature information and graph feature information are used as the image retrieval index corresponding to the driving image.

[0090] For example, after determining a knowledge graph from a driving image of a left-turn scene using a large language model, further feature extraction is performed on this knowledge graph to obtain graph feature information. When describing this graph feature information using a DSL (Domain Specific Language), the graph feature information can be:

[0091] SELECT*FROM table where scene=left-turn

[0092] MATCH(n:Car)-[rl:NearBack]-(o1:Truck)-[r2:NearHead]-(o2:Car),

[0093] (n:Car)-[r3:Head]-(o3:Pedestrain)-[r4:0n]-(o4:crosswalks)

[0094] WHERE n.angular.y>15AND ol.acc.x>0and o2.acc.x<0;”

[0095] In this embodiment, the driving image is first processed to obtain the image description information and target entities corresponding to the driving image. Then, based on the behavioral relationships between different target entities and the attribute information of each target entity, a knowledge graph corresponding to the driving image is determined. Subsequently, based on the image description information and the knowledge graph, an image retrieval index corresponding to the driving image is determined. In other words, the content of the driving image is indicated not only by the image description information but also by the knowledge graph. Since the knowledge graph indicates the target behavioral relationships and the attribute information of the target entities in the form of a graph, and the target behavioral relationships include the behavioral relationships between different target entities, the mined knowledge graph can more accurately indicate the key information in the driving image—the attribute information of the target entities and the driving relationships between different target entities. Therefore, combined with the image description information and the image retrieval index determined by the knowledge graph, the key information of the driving image can be indicated more accurately, the accuracy of the image retrieval index is higher, and the data mining effect of the driving image is better.

[0096] In some embodiments, the target image is retrieved from a database, which includes multiple driving images and their respective image retrieval indexes. If the process for determining the image retrieval index for each driving image is as described in S210-S230 above, S120 may further include: determining the text similarity between the retrieved text and the image description information of each driving image in the database; obtaining candidate driving images with text similarity higher than a first similarity threshold; determining the graph similarity between the retrieved knowledge graph and the knowledge graph of the candidate driving images; and determining the target image based on the candidate driving images with graph similarity higher than a second similarity threshold. The first and second similarity thresholds can be set according to requirements; for example, the first similarity threshold can be 0.7, and the second similarity threshold can be 0.75.

[0097] In this embodiment, the cosine similarity, Euclidean distance, or Manhattan distance between the retrieved text and the image description information of each driving image in the database can be used as the text similarity. Similarly, the cosine similarity, Euclidean distance, or Manhattan distance between the retrieved knowledge graph and the knowledge graph of the candidate driving images can be used as the graph similarity.

[0098] It can be either selecting the candidate driving images with the highest map similarity from those with map similarity higher than the second similarity threshold, or obtaining all candidate driving images with map similarity higher than the second similarity threshold and using them as the target image.

[0099] In this embodiment, the target image is retrieved by sequentially searching the text and the knowledge graph. Since the knowledge graph can indicate the attribute information of multiple entities and the relationship between the searched behavior, it can more accurately indicate the content of the search text. Therefore, when the target image is retrieved by combining the search text and the knowledge graph, a more accurate match is achieved, and the accuracy of the searched target image is improved.

[0100] In some embodiments, the image retrieval index for driving images includes textual feature information (obtained by feature extraction from the image description information of the driving images) and graph feature information (obtained by feature extraction from the knowledge graph corresponding to the driving images). Accordingly, S120 may include: determining the textual feature similarity between the feature extraction result of the retrieved text and the textual feature information of each driving image in the database; acquiring valid driving images with a textual feature similarity higher than a third similarity threshold; determining the graph feature similarity between the graph feature extraction result of the retrieved knowledge graph and the graph feature information of the valid driving images; and determining the target image based on the valid driving images with a graph feature similarity higher than a fourth similarity threshold. The third and fourth similarity thresholds can be set according to requirements; for example, the third similarity threshold can be 0.72, and the fourth similarity threshold can be 0.78.

[0101] Text feature similarity can be the cosine similarity, Euclidean distance, or Manhattan distance between the feature extraction results of the retrieved text and the text feature information of the driving image. Similarly, graph feature similarity can be the cosine similarity, Euclidean distance, or Manhattan distance between the graph feature extraction results of the retrieved knowledge graph and the graph feature information of the valid driving image.

[0102] The feature extraction results of the retrieved text can be determined by the text representation model, and the graph feature extraction results of the retrieved knowledge graph can be determined by the large language model.

[0103] In this embodiment, the search for images utilizes features extracted from the graph and features extracted from the text. Therefore, when matching the image description information of the retrieved text and the driving image, the similarity between the feature extraction results of the retrieved text and the text feature information of the driving image can more accurately indicate the differences between the image description information of the retrieved text and the driving image. Similarly, when matching the retrieved knowledge graph with valid driving images, the similarity between the graph feature extraction results of the retrieved knowledge graph and the graph feature information of the valid driving image can more accurately indicate the differences between the retrieved knowledge graph and the valid driving image. This achieves more precise matching and improves the accuracy of the searched target images.

[0104] In one embodiment, the image search process is as follows: Figure 4 As shown, it includes:

[0105] 401. Acquire driving video. This means acquiring the driving video to be processed, which was captured by the main vehicle.

[0106] 402. Traffic Scene Detection. This involves identifying whether the driving video includes a target traffic scene. If a target traffic scene is detected, the time window corresponding to the target traffic scene is determined based on the corresponding scene behavior video segment. Then, the video segment located within the time window corresponding to the target traffic scene is extracted from the driving video as the target video segment. Finally, video frames from the target video segment are obtained as driving images.

[0107] 403. Scene description information of the target traffic scene. After identifying the target traffic scene, obtain the scene description information of the target traffic scene.

[0108] 404. Add text descriptions to driving images. After obtaining the target traffic scene, the target prompt information corresponding to the target traffic scene can also be obtained. Then, based on the target prompt information corresponding to the target traffic scene, text descriptions are added to the driving images. Finally, based on the text descriptions added to the driving images and the scene description information of the target traffic scene, the image description information corresponding to the driving images is determined.

[0109] 405. Construction of the Knowledge Graph. Based on image description information and at least one of the driving images, determine the target behavior relationships (target behavior relationships include behavioral relationships between different target entities, and target entities include the vehicle that captured the driving images and traffic objects related to the vehicle's driving behavior). Then, based on the target behavior relationships and the attribute information of the target entities, a knowledge graph can be determined. The attribute information of the target entities can be collected by the vehicle's sensors, including radar, speed sensors, acceleration sensors, inertial navigation units, and cameras.

[0110] 406. Text Representation Model Processing. Text representation models are used to extract text features from image description information, yielding text feature information.

[0111] 407. Obtain text feature information.

[0112] 408. Determine the graph feature information. Extract graph features from the knowledge graph to obtain the graph feature information.

[0113] 409. Add to Database. Use textual feature information and map feature information as image retrieval indexes for vehicle images, and add the vehicle images and corresponding image retrieval indexes to the database to complete the data import.

[0114] 410. Image retrieval based on search text. A retrieval knowledge graph is constructed based on the search text. Then, the text feature similarity between the feature extraction results of the search text and the text feature information of each driving image in the database is determined. Valid driving images with text feature similarity higher than the third similarity threshold are obtained. The graph feature similarity between the graph feature extraction results of the retrieval knowledge graph and the graph feature information of the valid driving images is determined. Based on the valid driving images with graph feature similarity higher than the fourth similarity threshold, the target image is determined.

[0115] Of course, it is understandable that each video frame of the target video segment can be treated as a driving image to obtain a corresponding image retrieval index, and the obtained image retrieval index can be stored in the database.

[0116] After obtaining the search text, the target image can be retrieved from the database based on the search text in accordance with method 410.

[0117] Compared to traditional labeling methods, the method presented in this application possesses richer scene representation capabilities. It expresses scenes from two dimensions: temporal domain and entity spatial relationships, greatly amplifying the expressive power of scene mining. Simultaneously, by utilizing a database to store knowledge graphs and image description information, it supports image retrieval methods combining text and knowledge graphs, thereby improving retrieval accuracy.

[0118] See appendix Figure 5 , Figure 5 This illustration shows a structural block diagram of an image retrieval device according to an embodiment of this application, for use in an electronic device. The device 800 includes:

[0119] The acquisition module 810 is used to acquire the search text; the search text is used to describe the attribute information of multiple entities and the relationship between the searched behaviors; the multiple entities include the target vehicle and related search entities; the related search entities include traffic objects related to the driving behavior of the target vehicle; the relationship between the searched behaviors includes the relationship between behaviors between different entities;

[0120] The knowledge graph determination module 820 is used to construct a retrieval knowledge graph based on the retrieved text; the retrieval knowledge graph is used to indicate the relationship between the retrieved behavior and the attribute information of multiple entities;

[0121] The retrieval module 830 is used to perform image retrieval using the retrieval text and the retrieval knowledge graph as indexes, and to obtain the target image corresponding to the retrieval text.

[0122] Optionally, the graph determination module 820 is also used to determine the entity node corresponding to each entity based on the attribute information of each entity; for any two entities with behavioral association, node connection edges are constructed between the two entity nodes corresponding to the two entities based on the behavioral association between the two entities to obtain the retrieval knowledge graph.

[0123] Optionally, the target image is retrieved from a database, which includes multiple driving images and their respective image retrieval indexes. The device also includes a data processing module for acquiring image description information and target entities corresponding to the driving images. Target entities include the vehicle that captured the driving images and traffic objects related to the driving behavior of the vehicle. Based on the target behavior relationships and the attribute information of the target entities, a knowledge graph corresponding to the driving images is determined. Target behavior relationships include behavioral relationships between different target entities. The knowledge graph is used to indicate the target behavior relationships and the attribute information of the target entities. Based on the image description information and the knowledge graph corresponding to the driving images, an image retrieval index corresponding to the driving images is determined.

[0124] Optionally, the data processing module is also used to determine the target traffic scene where the main vehicle is located when the driving image is collected; determine the driver's reaction distance of the main vehicle's driver based on the target traffic scene; and obtain traffic objects whose distance from the main vehicle does not exceed the driver's reaction distance from traffic objects related to the main vehicle's driving behavior as target entities.

[0125] Optionally, the data processing module is also used to identify traffic scenes in the driving video to be processed; if a target traffic scene is identified in the driving video, the time window corresponding to the target traffic scene is determined based on the scene behavior video segment corresponding to the target traffic scene; wherein, the scene behavior video segment refers to the video segment in the driving video that includes the target driving behavior; the target driving behavior is the driving behavior performed by the main vehicle in response to the target traffic scene; the video segment located in the time window corresponding to the target traffic scene is obtained from the driving video as the target video segment; and the video frames in the target video segment are obtained as driving images.

[0126] Optionally, the data processing module is also used to add text descriptions to the driving image based on the target prompt information corresponding to the target traffic scene; and to determine the image description information corresponding to the driving image based on the text descriptions added to the driving image and the scene description information of the target traffic scene.

[0127] Optionally, the data processing module is also used to extract text features from the image description information to obtain text feature information; extract graph features from the knowledge graph to obtain graph feature information; and determine the image retrieval index corresponding to the driving image based on the text feature information and the graph feature information.

[0128] Optionally, the data processing module is also used to determine the behavioral association between different target entities based on at least one of the image description information and the driving image, so as to obtain the target behavioral association.

[0129] Optionally, the retrieval module 830 is further configured to determine the text similarity between the retrieved text and the image description information of each driving image in the database; obtain candidate driving images with text similarity higher than a first similarity threshold; determine the graph similarity between the retrieved knowledge graph and the knowledge graph of the candidate driving images; and determine the target image based on the candidate driving images with graph similarity higher than a second similarity threshold.

[0130] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0131] Furthermore, the functions in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module.

[0132] Please refer to Figure 6This document illustrates a structural block diagram of an electronic device 500 according to an embodiment of this application. The electronic device 500 can be a smartphone, tablet computer, e-reader, vehicle, or other electronic device capable of running applications. The electronic device 500 in this application may include one or more of the following components: a processor 510, a memory 520, and one or more applications. One or more applications may be stored in the memory 520 and configured to be executed by one or more processors 510, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0133] Processor 510 may include one or more processing cores. Processor 510 connects to various parts within the electronic device 500 using various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing instructions, programs, code sets, or instruction sets stored in memory 520, and by calling data stored in memory 520. Optionally, processor 510 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 510 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 510 and may be implemented separately using a communication chip.

[0134] The memory 520 may include random access memory (RAM) or read-only memory (ROM). The memory 520 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 520 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 500 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0135] Furthermore, the functions in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module.

[0136] On the other hand, this application also provides a computer-readable storage medium storing program code that can be called by a processor to execute the methods described in the above method embodiments.

[0137] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or a cluster of ROMs. Optionally, computer-readable storage media include non-transitory computer-readable storage media. The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image retrieval method, characterized in that, The method includes: Obtain the search text; the search text is used to describe the attribute information of multiple entities and the relationship between the searched behaviors; the multiple entities include the target vehicle and related search entities; the related search entities include traffic objects related to the driving behavior of the target vehicle; the relationship between the searched behaviors includes the relationship between behaviors between different entities; A retrieval knowledge graph is constructed based on the retrieved text; the retrieval knowledge graph is used to indicate the association relationship of the retrieved behavior and the attribute information of the multiple entities; Image retrieval is performed using the search text and the search knowledge graph as indexes to obtain the target image corresponding to the search text.

2. The method according to claim 1, characterized in that, The construction of the retrieval knowledge graph based on the retrieved text includes: Based on the attribute information of each entity, determine the entity node corresponding to each entity; For any two entities with a behavioral relationship, node connection edges are constructed between the two entity nodes corresponding to the two entities based on the behavioral relationship between the two entities to obtain the retrieval knowledge graph.

3. The method according to claim 1, characterized in that, The target image is retrieved from a database, which includes multiple driving images and image retrieval indexes for each driving image. Before performing image retrieval using the search text and the search knowledge graph as indexes to obtain the target image corresponding to the search text, the method further includes: Obtain image description information and target entities corresponding to the driving images; the target entities include the vehicle that captured the driving images and traffic objects related to the driving behavior of the vehicle. Based on the target behavior associations and the attribute information of the target entities, a knowledge graph corresponding to the driving image is determined; the target behavior associations include the behavior associations between different target entities; the knowledge graph is used to indicate the target behavior associations and the attribute information of the target entities. Based on the image description information and knowledge graph corresponding to the driving image, the image retrieval index corresponding to the driving image is determined.

4. The method according to claim 3, characterized in that, Before obtaining the image description information and target entity corresponding to the driving image, the method further includes: Determine the target traffic scene where the main vehicle is located when the driving images are acquired; Determine the driver's reaction distance of the main vehicle's driver based on the target traffic scenario; From the traffic objects related to the driving behavior of the main vehicle, obtain the traffic objects whose distance from the main vehicle does not exceed the driver's reaction distance as target entities.

5. The method according to claim 3, characterized in that, Before acquiring the image description information and target entity corresponding to the driving image, the method further includes: Traffic scene recognition is performed on the driving videos to be processed; If a target traffic scene is identified in the driving video, a time window corresponding to the target traffic scene is determined based on the scene behavior video segment corresponding to the target traffic scene; wherein, the scene behavior video segment refers to a video segment in the driving video that includes the target driving behavior; the target driving behavior is the driving behavior performed by the main vehicle in response to the target traffic scene; The video segment located within the time window corresponding to the target traffic scene is obtained from the driving video and used as the target video segment; The video frames in the target video segment are obtained as the driving images.

6. The method according to claim 3, characterized in that, Before acquiring the image description information and target entity corresponding to the driving image, the method further includes: Based on the target prompt information corresponding to the target traffic scene, add text descriptions to the driving image; Based on the text description added to the driving image and the scene description information of the target traffic scene, the image description information corresponding to the driving image is determined.

7. The method according to claim 3, characterized in that, The step of determining the image retrieval index corresponding to the driving image based on the image description information and knowledge graph corresponding to the driving image includes: Text features are extracted from the image description information to obtain text feature information; The knowledge graph is subjected to graph feature extraction to obtain graph feature information; Based on the text feature information and the map feature information, the image retrieval index corresponding to the driving image is determined.

8. The method according to claim 3, characterized in that, Before determining the knowledge graph corresponding to the driving image based on the target behavior association and the attribute information of the target entity, the method further includes: Based on the image description information and at least one of the driving images, the behavioral associations between different target entities are determined to obtain the target behavioral associations.

9. The method according to claim 3, characterized in that, The step of performing image retrieval using the search text and the search knowledge graph as indexes to obtain the target image corresponding to the search text includes: Determine the text similarity between the retrieved text and the image description information of each driving image in the database; Obtain candidate vehicle images with text similarity higher than the first similarity threshold; Determine the graph similarity between the retrieved knowledge graph and the knowledge graph of the candidate driving images; The target image is determined based on candidate driving images whose map similarity is higher than a second similarity threshold.

10. An image retrieval device, characterized in that, The device includes: The acquisition module is used to acquire search text; the search text is used to describe the attribute information of multiple entities and the relationship between the searched behaviors; the multiple entities include the target vehicle and related search entities; the related search entities include traffic objects related to the driving behavior of the target vehicle; the relationship between the searched behaviors includes the relationship between behaviors between different entities; The knowledge graph determination module is used to construct a retrieval knowledge graph based on the retrieval text; the retrieval knowledge graph is used to indicate the association relationship of the searched behavior and the attribute information of the multiple entities; The retrieval module is used to perform image retrieval using the retrieval text and the retrieval knowledge graph as indexes, and obtain the target image corresponding to the retrieval text.

11. An electronic device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores processor-executable program code, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1-9.

Images