Scene library construction and retrieval method, system and medium for multi-dimensional feature coding

Abstract

The present application discloses a method, system and medium for constructing and retrieving a scene library with multi-dimensional feature coding, which relates to the field of autonomous driving. The method includes: constructing a hierarchical model; quantifying the feature information of different scene elements in each information layer to obtain the feature coding of the scene elements in each information layer; multi-dimensionally combining the feature coding of the scene elements of all information layers to obtain the standard coding corresponding to the vehicle driving scene; constructing a scene library based on the original data, feature information, feature coding and standard coding of different vehicle driving scenes; based on the standard coding of the target driving scene and / or the feature coding of the target scene element, retrieving the standard coding stored in the scene library, and obtaining the original data of the corresponding vehicle driving scene from the scene library. The present application fully considers the multi-facetedness and correlation of different scene elements in the process of scene library construction, and improves the accuracy of target driving scene retrieval.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method, system and medium for constructing and retrieving a scene library of multi-dimensional feature coding. Background Art

[0002] With the rapid development of the intelligent era, the technology of autonomous driving has gradually entered the public's field of vision and has become the primary selling point of more and more car brands. In order to make the autonomous driving of cars have a higher sense of experience, the requirements for the autonomous driving systems they are equipped with are also getting higher and higher.

[0003] A complete and high-performance autonomous driving system usually relies on multimodal data of different vehicle driving scenarios for training, testing and optimization. The existence of the scenario library provides an effective storage, management and access space for these multimodal data. However, in the construction process of the existing scenario library, more attention is often paid to the single-dimensional characteristics of different vehicle driving scenarios, that is, the overall data of the vehicle driving scenario is marked and stored, and the impact of different scenario factors on the entire vehicle driving scenario is not considered. Therefore, the description of the vehicle driving scenario is not comprehensive.

[0004] In addition, the retrieval method of the existing scene library is relatively simple. Usually, it can only perform overall retrieval of vehicle driving scenes based on labels or codes, and cannot perform targeted retrieval of vehicle driving scenes containing specific scene elements. This also leads to the low accuracy of the retrieval of the existing scene library. Summary of the invention

[0005] The purpose of this application is to provide a method, system and medium for constructing and retrieving a scene library with multi-dimensional feature coding, which fully considers the multi-facetedness and correlation of different scene elements in the process of scene library construction, making the description of vehicle driving scenes more comprehensive, while also improving the accuracy of target driving scene retrieval.

[0006] To achieve the above objectives, this application provides the following solutions.

[0007] In the first aspect, the present application provides a method for constructing and retrieving a scene library of multi-dimensional feature coding, the method comprising: constructing a hierarchical model; the hierarchical model comprising multiple information layers; each information layer is used to store original data of different scene elements in a vehicle driving scene; extracting feature information of different scene elements in each information layer respectively; quantizing the feature information of different scene elements in each information layer to obtain feature codes of scene elements in each information layer; multi-dimensionally combining the feature codes of scene elements of all information layers to obtain a standard code corresponding to the vehicle driving scene; each bit of the standard code corresponds to a feature code of a scene element; based on the original data of different vehicle driving scenes, different vehicle driving scenes The method comprises the following steps: constructing a scene library by combining characteristic information of a target scene, characteristic codes of different vehicle driving scenes, and standard codes of different vehicle driving scenes; the original data of the vehicle driving scene includes the original data of multiple scene elements; the characteristic information of the vehicle driving scene includes the characteristic information of multiple scene elements; the characteristic codes of the vehicle driving scene include the characteristic codes of multiple scene elements; based on the standard code of the target driving scene, directly retrieving the standard code stored in the scene library, and obtaining the original data of the vehicle driving scene corresponding to the target driving scene from the scene library, and / or, based on the characteristic code of the target scene element, bit-by-bit retrieving the standard code stored in the scene library, and obtaining the original data of the vehicle driving scene containing the target scene element from the scene library.

[0008] In a second aspect, the present application also provides a computer system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-dimensional feature coding scene library construction and retrieval method described in the first aspect.

[0009] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the scene library construction and retrieval method for multi-dimensional feature coding described in the second aspect.

[0010] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0011] First, this application constructs a hierarchical model and uses different information layers to divide different scene elements in the vehicle driving scene, so that the vehicle driving scene has multi-level features, and the features of each level can be represented independently and can also be related to each other. Secondly, this application also proposes the concept of standard code (SC), and realizes the multi-dimensional feature coding of scene elements by multi-dimensionally combining the feature coding of scene elements in each information layer, and gives each vehicle driving scene its corresponding standard coding, which is convenient for the accurate retrieval of the subsequent scene library. Finally, since the standard coding of this application is composed of feature coding of different scene elements, in addition to searching the standard coding as a whole, one or several bits in the standard coding can also be searched separately, and this retrieval method accurate to the scene element effectively guarantees the accuracy of the scene library retrieval. Therefore, this application fully considers the multi-faceted and correlation of different scene elements in the process of scene library construction, making the description of vehicle driving scenes more comprehensive, and also improving the accuracy of target driving scene retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0013] Figure 1 A flowchart of a method for constructing and retrieving a scene library with multi-dimensional feature coding provided in an embodiment of the present application.

[0014] Figure 2 This is a diagram of the internal structure of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] The purpose of this application is to provide a multi-dimensional feature-coded scene library construction and retrieval method, system and medium, which fully considers the multi-faceted and correlation of different scene elements in the scene library construction process, making the description of vehicle driving scenes more comprehensive, while also improving the accuracy of target driving scene retrieval.

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] Example 1

[0019] This embodiment provides a method for constructing and retrieving a scene library with multi-dimensional feature coding, such as Figure 1 As shown, the scene library construction and retrieval method of the multi-dimensional feature coding is specifically as follows.

[0020] Step S1: construct a hierarchical model; the hierarchical model includes multiple information layers; each information layer is used to store original data of different scene elements in a vehicle driving scene.

[0021] In this embodiment, the multiple information layers of the hierarchical model are road layout layer, terrain attribute layer, temporary change layer, interactive member layer and environmental factor layer; the categories of scene elements include: road geometry, road topology, road surface attributes, road terrain geometry, road change information, road obstacles, road traffic participants and road environment.

[0022] Specifically, the road layout layer is used to store the original data of the road geometry and the original data of the road topology; the terrain attribute layer is used to store the original data of the road surface attributes and the original data of the road terrain geometry; the temporary change layer is used to store the original data of the road change information; the interactive member layer is used to store the original data of the road obstacles and the original data of the road traffic participants; the environmental factor layer is used to store the original data of the road environment. The scene elements contained in each information layer are specifically shown in the table below.

[0023] Table 1 Scene elements contained in each information layer

[0024]

[0025] Furthermore, each information layer of the hierarchical model has a dynamic update mechanism, which is used to update the original data of different scene elements in each information layer in real time. For example, the road layout layer is initialized and dynamically updated using high-precision map data, and the terrain attribute layer, temporary change layer, interactive member layer, and environmental factor layer are dynamically updated using data collected by different sensors (such as lidar, cameras, etc.) to achieve dynamic capture of the time-varying nature of vehicle driving scenes.

[0026] Step S2: extracting feature information of different scene elements in each information layer respectively.

[0027] In this embodiment, since the data collected by some sensors cannot be used directly, it is necessary to extract feature information thereof.

[0028] In one example, the road layout layer: the lane layout is extracted through image processing technology, and then the median filter, boundary filter, Sobel operator and other algorithms are applied to process the roughness map to extract the number of road boundaries. , the curvature of the road boundary .

[0029] Terrain attribute layer: Based on the roughness map and height map, the average roughness of the ground is calculated , average height from ground level .

[0030] Temporary change layer: sensors detect road construction and record the type of change Construction, duration Hours, construction location .

[0031] Interaction member layer: Identify dynamic traffic participants, such as Pedestrians and participants’ positions , participant speed , participation size Medium, and predict its trajectory.

[0032] Environmental factors layer: sensors monitor current weather Sunny, sunny conditions Good, sight distance .

[0033] Step S3: quantizing the feature information of different scene elements in each information layer to obtain feature codes of the scene elements in each information layer.

[0034] In this embodiment, the feature information of different scene elements in each information layer is quantized. Specifically, the feature encoding of specific scene elements in different vehicle driving scenes is first performed, and the distribution of the feature encoding is statistically analyzed, and then the scaling factor (empirical parameter) and movement factor (empirical parameter) of the different scene elements are determined. Then, based on the scaling factor and movement factor of the different scene elements, the feature information of different scene elements in each information layer is scaled and linearly moved. The process of quantization is to scale and convert floating point values ​​into integer values, and ensure that each feature has sufficient discrimination in different vehicle driving scenes. Among them, the feature encoding of the scene elements in each information layer is as follows.

[0035] .

[0036] In the formula, For the i Feature encoding of scene elements in each information layer, For the iThe feature encoding of the first scene element in the information layer, For the i The feature encoding of the second scene element in the information layer, For the i In the information layer n Feature encoding of scene elements.

[0037] In an example, the road layout layer: number of boundaries , after scaling and linear shifting, quantized to 5; the boundary curvature , after scaling and linear shifting, it is quantized to 3; the feature encoding of the scene elements in the road layout layer is {5, 3}.

[0038] Terrain attribute layer: average ground roughness , after scaling and linear shifting, quantized to 6; average ground height , after scaling and linear shifting, it is quantized to 9; the feature encoding of the scene elements in the terrain attribute layer is {6, 9}.

[0039] Temporary Change Layer: Change Type Construction, assigned to specific category code 4; duration Hours, after scaling and linear shifting, quantized to 8; construction position , after scaling and linear shifting, it is quantized to 5; the feature encoding of the scene elements in the temporary change layer is {4, 8, 5}.

[0040] Interactive member layer: dynamic traffic participants Pedestrians, assigned a specific category code 2; participant speed , quantized to 1; participation size Medium, quantized to 5; predicted trajectory, quantized to 7; the feature encoding of scene elements in the interaction member layer is {2, 1, 5, 7}.

[0041] Environmental factors layer: weather of the day Sunny, quantized as 2; lighting conditions Good, quantized as 9; viewing distance , quantized to 8; the feature encoding of the scene elements in the environmental factor layer is {2, 9, 8}.

[0042] Step S4: Combining the feature codes of the scene elements of all information layers in multiple dimensions to obtain a standard code corresponding to the vehicle driving scene; each bit of the standard code corresponds to a feature code of a scene element.

[0043] In this embodiment, the standard encoding is specifically as follows.

[0044] .

[0045] In the formula, SC For standard encoding, Characteristic encoding of scene elements in the first information layer, Characteristic encoding of scene elements in the second information layer, For the m Although the standard code corresponding to the vehicle driving scene is composed of the feature codes of multiple scene elements, the standard code actually obtained may correspond to only one vehicle driving scene or may correspond to multiple vehicle driving scenes at the same time.

[0046] In one example, the feature codes of scene elements of all information layers are combined in multiple dimensions to obtain the following standard code.

[0047] SC ={5, 3, 6, 9, 4, 8, 5, 2, 1, 5, 7, 2, 9, 8}.

[0048] Step S5: constructing a scenario library based on original data of different vehicle driving scenarios, characteristic information of different vehicle driving scenarios, characteristic codes of different vehicle driving scenarios, and standard codes of different vehicle driving scenarios; the original data of the vehicle driving scenarios include original data of multiple scenario elements; the characteristic information of the vehicle driving scenarios include characteristic information of multiple scenario elements; the characteristic codes of the vehicle driving scenarios include characteristic codes of multiple scenario elements.

[0049] In this embodiment, the standard code SC is a complete code string, which is stored in the scene library under the corresponding code category (Category Code, CC), and the standard codes of different vehicle driving scenes are stored by category.

[0050] Furthermore, the dynamic expansion of the scenario library refers to the automatic expansion mechanism of the scenario library, which can adapt to the changes of the newly added scenarios. When a new vehicle driving scenario is generated, the system first calculates its SC, and then indexes the storage path in the scenario library step by step according to the number of bits of the SC. If the SC is a new combination, the system will automatically create a new CC for it and store the original data of the new vehicle driving scenario under the CC, thereby realizing the dynamic update and expansion of the scenario library.

[0051] Step S6: Based on the standard code of the target driving scene, the standard code stored in the scene library is directly retrieved, and the original data of the vehicle driving scene corresponding to the target driving scene is obtained from the scene library, and / or, based on the feature code of the target scene element, the standard code stored in the scene library is bit-by-bit retrieved, and the original data of the vehicle driving scene containing the target scene element is obtained from the scene library.

[0052] In this embodiment, the fast search based on the number of SC bits is a method of quickly locating a specific vehicle driving scene by decoding the SC bit by bit. For example, the scene library can be quickly classified and searched through the first few bits of the SC, so as to efficiently find the vehicle driving scenes under specific road layouts, terrain attributes, and environmental conditions, greatly improving the search efficiency. The search of the existing scene library can only search the vehicle driving scene as a whole, and cannot search it by querying specific scene elements. Both methods of the present application can be achieved.

[0053] In summary, this embodiment can comprehensively and accurately describe the multi-level features in the autonomous driving scene, which can not only meet the expression needs of road layout, dynamic changes, traffic participants and environmental factors in the urban environment, but also effectively deal with irregular off-road scenes, covering the scene characteristics under various complex driving conditions.

[0054] Secondly, the scene library in this embodiment realizes the refined management of scene features through hierarchical models and multi-dimensional feature coding, so that the features of each layer can be represented independently and can be interrelated, satisfying the real-time capture of dynamic changes and complex interactions of vehicle driving scenes. At the same time, the scene organization and coding based on SC enables different scene types to be effectively distinguished, and fast retrieval and efficient analysis can be achieved through coding indexes. The SC and multi-layer indexing mechanism significantly improves the organization, scalability and retrieval efficiency of the scene library, can quickly respond to query requirements in large-scale scene libraries, and provide comprehensive and reliable data support for autonomous driving decisions in complex scenes.

[0055] Example 2

[0056] This embodiment provides a computer system, which may be a server or a terminal, and its internal structure diagram may be as follows: Figure 2 As shown. The computer system includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer system is used to provide computing and control capabilities. The memory of the computer system includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer system is used to exchange information between the processor and an external device. The communication interface of the computer system is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a scene library construction and retrieval method for multi-dimensional feature coding is implemented.

[0057] Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer system to which the scheme of the present application is applied. The specific computer system may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0058] Example 3

[0059] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0061] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0062] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0063] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0064] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for constructing and retrieving a scene library with multi-dimensional feature coding, characterized in that: The multi-dimensional feature coding scene library construction and retrieval method includes: Constructing a hierarchical model; the hierarchical model includes a plurality of information layers; each information layer is used to store raw data of different scene elements in a vehicle driving scene; Extract the characteristic information of different scene elements in each information layer respectively; Quantifying the feature information of different scene elements in each information layer to obtain the feature codes of the scene elements in each information layer; The characteristic codes of the scene elements of all information layers are multi-dimensionally combined to obtain a standard code corresponding to the vehicle driving scene; each bit of the standard code corresponds to a characteristic code of a scene element; A scenario library is constructed based on raw data of different vehicle driving scenarios, feature information of different vehicle driving scenarios, feature codes of different vehicle driving scenarios, and standard codes of different vehicle driving scenarios; the raw data of vehicle driving scenarios include raw data of multiple scenario elements; the feature information of vehicle driving scenarios include feature information of multiple scenario elements; the feature codes of vehicle driving scenarios include feature codes of multiple scenario elements; Based on the standard coding of the target driving scene, the standard coding stored in the scene library is directly retrieved, and the original data of the vehicle driving scene corresponding to the target driving scene is obtained from the scene library, and / or, based on the feature coding of the target scene element, the standard coding stored in the scene library is bit-by-bit retrieved, and the original data of the vehicle driving scene containing the target scene element is obtained from the scene library.

2. The method for constructing and retrieving a scene library of multi-dimensional feature coding according to claim 1, characterized in that: The categories of scene elements include: road geometry, road topology, road surface attributes, road terrain geometry, road change information, road obstacles, road traffic participants and road environment.

3. The method for constructing and retrieving a scene library of multi-dimensional feature coding according to claim 2, characterized in that: The multiple information layers of the hierarchical model are a road layout layer, a terrain attribute layer, a temporary change layer, an interactive member layer and an environmental factor layer; The road layout layer is used to store the original data of the road geometry and the original data of the road topology; the terrain attribute layer is used to store the original data of the road surface attributes and the original data of the road terrain geometry; the temporary change layer is used to store the original data of the road change information; the interactive member layer is used to store the original data of the road obstacles and the original data of the road traffic participants; the environmental factor layer is used to store the original data of the road environment.

4. The method for constructing and retrieving a scene library of multi-dimensional feature coding according to claim 1, characterized in that: Each information layer of the hierarchical model has a dynamic update mechanism; the dynamic update mechanism is used to update the original data of different scene elements in each information layer in real time.

5. The method for constructing and retrieving a scene library of multi-dimensional feature coding according to claim 1, characterized in that: The characteristic information of different scene elements in each information layer is quantified, including: Determine the scaling and movement factors of different scene elements; Based on the scaling factors and movement factors of different scene elements, the feature information of different scene elements in each information layer is scaled and linearly moved.

6. The method for constructing and retrieving a scene library of multi-dimensional feature coding according to claim 1, characterized in that: The feature encoding of scene elements in each information layer is: ； In the formula, For the i Feature encoding of scene elements in each information layer, For the i The feature encoding of the first scene element in the information layer, For the i The feature encoding of the second scene element in the information layer, For the i In the information layer n Feature encoding of scene elements.

7. The method for constructing and retrieving a scene library of multi-dimensional feature coding according to claim 1, characterized in that: The standard code is: ； In the formula, SC For standard encoding, Characteristic encoding of scene elements in the first information layer, Characteristic encoding of scene elements in the second information layer, For the m Feature encoding of scene elements at each information layer.

8. The method for constructing and retrieving a scene library of multi-dimensional feature coding according to claim 1, characterized in that: The standard codes for different vehicle driving scenarios in the scenario library are stored according to the coding categories; one category of standard codes corresponds to one or more vehicle driving scenarios.

9. A computer system comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-dimensional feature coding scene library construction and retrieval method described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing and retrieving a scene library of multi-dimensional feature coding described in any one of claims 1 to 8 is implemented.

Images