Automatic driving scene mining method and device, equipment, medium and product

Abstract

The invention discloses an automatic driving scene mining method and device, equipment, a medium and a product. The method comprises the steps that if a mining instruction of a user is received, at least one target scene element is extracted from the mining instruction based on a language processing model; determining a scene mining result in a pre-established scene knowledge base according to the target scene element; wherein the scene knowledge base comprises a plurality of scene codes, and each scene code comprises a plurality of scene element codes; the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image. According to the technical scheme, the problem that automatic driving scene mining is poor in flexibility is solved, based on fusion and question and answer interaction of multi-modal scene data, the recognition capability on new scenes and complex scenes is effectively improved, dependence on manpower and professional knowledge is reduced, and intelligence and consistency of the mining process are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, medium and product for mining autonomous driving scenarios. Background Art

[0002] Autonomous driving tests require a large number of driving scenarios as support. Usually, driving scenarios come from staff's analysis of the driving functions of autonomous vehicles, such as the ODD (Operational Design Domain) range in which driving functions are effective. However, due to individual cognitive biases or blind spots in the scene design of staff, all scenarios facing the function cannot be fully considered. Therefore, a major source of driving scenarios is real data collection. Since the probability of key scenarios is low and there are many types, although they are not common, once they occur, they are likely to cause serious safety problems. The distribution of key scenarios in the collected data shows a long tail phenomenon. Therefore, how to extract key scenarios from the collected data has become an important issue that limits the testing capabilities of autonomous driving vehicles, that is, the scene mining problem.

[0003] At present, the mining methods of driving scenes in the field of autonomous driving mainly include vision-based semantic matching retrieval methods and expert experience-based scene understanding methods. The vision-based semantic matching retrieval method relies on predefined rules, which leads to its lack of adaptability and poor flexibility when dealing with unforeseen or complex scenes. The scene understanding method based on expert experience relies on manual judgment, which limits the efficiency of large-scale scene mining, and the mining results are subjective. Summary of the invention

[0004] The present invention provides a method, device, equipment, medium and product for mining autonomous driving scenes to solve the problem of poor flexibility in autonomous driving scene mining. Based on the fusion of multimodal scene data and question-answer interaction, it effectively improves the recognition ability of new and complex scenes, reduces dependence on manual and professional knowledge, and enhances the intelligence and consistency of the mining process.

[0005] According to one aspect of the present invention, a method for mining an autonomous driving scenario is provided, the method comprising:

[0006] If a mining instruction from a user is received, extracting at least one target scene element from the mining instruction based on a language processing model;

[0007] Determining a scene mining result in a pre-established scene knowledge base according to the target scene elements;

[0008] The scene knowledge base includes a plurality of scene codes, each of which includes a plurality of scene element codes; the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image.

[0009] According to another aspect of the present invention, there is provided a mining device for an autonomous driving scenario, the device comprising:

[0010] A scene element extraction module, configured to extract at least one target scene element from the mining instruction based on a language processing model upon receiving a mining instruction from the user;

[0011] A mining result determination module, used to determine the scene mining result in a pre-established scene knowledge base according to the target scene elements;

[0012] The scene knowledge base includes a plurality of scene codes, each of which includes a plurality of scene element codes; the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for mining autonomous driving scenarios described in any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for mining autonomous driving scenarios described in any embodiment of the present invention when executed.

[0016] According to another aspect of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method for mining autonomous driving scenarios described in any embodiment of the present invention.

[0017] The technical solution of the embodiment of the present invention, if a mining instruction from a user is received, then based on a language processing model, at least one target scene element is extracted from the mining instruction; based on the target scene element, the scene mining result is determined in a pre-established scene knowledge base; wherein the scene knowledge base includes multiple scene codes, each scene code includes multiple scene element codes; the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image. This technical solution solves the problem of poor flexibility in autonomous driving scene mining, and based on the fusion of multimodal scene data and question-answer interaction, effectively improves the recognition ability of new and complex scenes, reduces dependence on manual and professional knowledge, and enhances the intelligence and consistency of the mining process.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flowchart of a method for mining an autonomous driving scenario provided according to Embodiment 1 of the present invention;

[0021] Figure 2 is a flowchart of a method for mining autonomous driving scenarios according to Embodiment 2 of the present invention;

[0022] Figure 3 is a schematic diagram of the structure of an excavation device for an autonomous driving scenario provided according to a third embodiment of the present invention;

[0023] Figure 4 It is a schematic diagram of the structure of an electronic device that implements the method for mining autonomous driving scenarios according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0026] Embodiment 1

[0027] Figure 1 A flowchart of a method for mining an autonomous driving scene is provided for the first embodiment of the present invention. This embodiment is applicable to autonomous driving scenes, especially the processing of autonomous driving information. The method can be executed by a mining device for autonomous driving scenes, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0028] S110: If a mining instruction from the user is received, extract at least one target scene element from the mining instruction based on a language processing model.

[0029] This solution can be executed by a test platform for an autonomous driving vehicle, and the test platform can perform multiple driving scenario tests on the autonomous driving vehicle. The test platform can configure a user terminal to interact with the user. The mining instruction can be a search text entered by the user in the search box of the user terminal, or a scene search option clicked by the user, or a voice instruction issued by the user. The user terminal can obtain the user's mining instruction and send the mining instruction to the test platform. After receiving the mining instruction sent by the user terminal, the test platform can parse the mining instruction based on the language processing model and extract the target scene elements in the mining instruction.

[0030] Among them, the language processing model can be a natural language processing (NLP) model such as GPT (Generative Pre-trained Transformer), BERT (Bidirectional Encoder Representations from Transformers). Mining instructions can be used to represent the user's constraints on the target scene, and the mining instructions can include one or more scene elements. The scene element can be a description of the target in the driving scene, and the target can be a person or an object. For example, the scene element can be the number of lanes, the direction of lane travel, the degree of pedestrian congestion, the degree of vehicle congestion, the color of traffic lights, and the location of traffic guardrails.

[0031] S120. Determine a scene mining result in a pre-established scene knowledge base according to the target scene element; wherein the scene knowledge base includes a plurality of scene codes, each scene code includes a plurality of scene element codes; and the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image.

[0032] The test platform can search in the scene knowledge base according to the target scene elements extracted in the mining instruction, and determine the scene mining results based on the search results. Specifically, the test platform can generate a scene code matching the driving scene image based on the driving scene image and the scene description text corresponding to the driving scene image. It can be understood that a scene code can represent a driving scene. Based on the scene codes corresponding to multiple driving scene images, the test platform can form a scene knowledge base. Each scene code can be composed of multiple scene element codes.

[0033] The test platform can encode the target scene elements, match the target scene element encoding with the scene element encoding in each scene encoding in the scene knowledge base, and generate a scene mining result according to the matching result. The scene mining result can be used to represent the search result of the target scene. The scene mining result can be a driving scene that is the same as the target scene, or a preset number of driving scenes that have a high degree of similarity with the target scene.

[0034] The technical solution of the embodiment of the present invention, if a mining instruction from a user is received, then based on a language processing model, at least one target scene element is extracted from the mining instruction; based on the target scene element, the scene mining result is determined in a pre-established scene knowledge base; wherein the scene knowledge base includes multiple scene codes, each scene code includes multiple scene element codes; the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image. This technical solution solves the problem of poor flexibility in autonomous driving scene mining, and based on the fusion of multimodal scene data and question-answer interaction, effectively improves the recognition ability of new and complex scenes, reduces dependence on manual and professional knowledge, and enhances the intelligence and consistency of the mining process.

[0035] Embodiment 2

[0036] Figure 2 This is a flow chart of a method for mining an autonomous driving scene provided by the second embodiment of the present invention. This embodiment is based on the above embodiment and refines the process of building a scene knowledge base. Figure 2 As shown, the method includes:

[0037] S210: Acquire a driving scene image, perform visual perception processing on the driving scene image, and output a visual perception result.

[0038] In this solution, the visual perception processing includes at least one of target detection, target segmentation and target recognition; the targets are people and objects in the driving scene.

[0039] The test platform can obtain driving scene images from a shooting device mounted on a vehicle, or from a monitoring device mounted around a road. In a feasible solution, the test platform can extract driving scene images from a monitoring video according to a preset extraction principle. After obtaining the driving scene image, the test platform can perform visual perception processing such as target detection, target segmentation, and target recognition on people or objects in the driving scene image to obtain a visual perception result. The visual perception result may include information such as detection results, segmentation results, and recognition results. Among them, visual perception processing such as target detection, target segmentation, and target recognition may be implemented based on a deep learning algorithm. The visual perception result may include one or more of a target detection result, a target segmentation result, or a target recognition result. In a specific example, the visual perception result may include information such as lane line position, lane line type, pedestrian position, vehicle position, vehicle type, signal light position, signal light indication direction, and signal light color.

[0040] S220: Generate a scene description text matching the driving scene image according to the visual perception result.

[0041] It is understandable that different driving scene images have different contents and contain different targets. The test platform can generate scene description texts corresponding to driving scene images according to the visual perception results of driving scene images and the preset scene description template to describe different driving scene images in a standardized manner. Among them, the scene description text can describe the driving scene image in the form of text. The scene description template can be a description method of the driving scene image, and can include constraint information such as the description order, description angle, and description format of the scene elements. For example, the scene description template can be that there are a total of ×× lanes in driving scene image 1, among which ×× can be changed lanes and ×× cannot be changed lanes; there are a total of ×× vehicles in lane 1, among which ×× trucks, ×× cars, ×× buses, and ×× motorcycles; there are a total of ×× vehicles in lane 2...

[0042] The test platform can further process the visual perception results and generate scene description text based on the data processing results. For example, the test platform can count the number of vehicles based on the vehicle detection results, and determine the correspondence between traffic lights and lanes based on the lane line recognition results and traffic light detection results.

[0043] S230: Generate a scene code according to the driving scene image and the scene description text matched with the driving scene image.

[0044] It is easy to understand that the driving scene image can represent the driving scene from the visual image mode, and the scene description text can represent the driving scene from the text mode. The test platform can generate a scene encoding that integrates multi-modal data features based on the driving scene image and the scene description text matched with the driving scene image, and more fully mine the scene description information.

[0045] In a feasible solution, generating a scene code according to the driving scene image and a scene description text matched with the driving scene image includes:

[0046] A scene code is generated according to the driving scene image and the scene description text matched with the driving scene image based on a visual language fusion model.

[0047] In this solution, the test platform can perform synchronous data processing on the driving scene image and its scene description text based on the visual language fusion model to generate a scene encoding that matches the driving scene image. The visual language fusion model can be used to embed multimodal information into the same semantic space, such as the CLIP (Contrastive Language-Image Pre-training) model. The CLIP model consists of two parts: an image encoder and a text encoder. The image encoder is used to convert the image into a feature vector, which can be a convolutional neural network (such as ResNet) or a Transformer model (such as ViT). The text encoder is responsible for converting the text into a feature vector, which is usually a Transformer model. The image encoder and the text encoder share a vector space to achieve cross-modal information interaction and fusion.

[0048] Based on the above scheme, the visual language fusion model includes an image encoder, a text encoder and a feature fuser;

[0049] The step of generating a scene code based on the driving scene image and the scene description text matched with the driving scene image and based on a visual language fusion model includes:

[0050] Inputting the driving scene image into the image encoder to obtain image features, and inputting the scene description text matched with the driving scene image into the text encoder to obtain text features;

[0051] Inputting the image features and the text features into the feature fusion device to obtain multiple scene element codes;

[0052] Multiple scene element codes are combined to obtain a scene code.

[0053] The test platform can input the driving scene image into the image encoder, the image encoder extracts features from the driving scene image to obtain image features, and inputs the scene description text matched with the driving scene image into the text encoder, the text encoder extracts features from the scene description text to obtain text features. The test platform can input the image features and text features into the feature fusion device, the feature fusion device can perform feature fusion and feature correspondence on the image features and text features to obtain multiple scene element codes, and combine the multiple scene element codes to obtain the scene code corresponding to the driving scene image.

[0054] S240: Establish a scene knowledge base according to the scene codes matched by each driving scene image in the scene sample set.

[0055] The test platform may process all driving scene images in the scene sample set acquired in advance according to S210 - S230 to obtain scene codes matched with each driving scene image, so as to establish a scene knowledge base.

[0056] S250: If a mining instruction from the user is received, extract at least one target scene element from the mining instruction based on a language processing model.

[0057] S260. Determine a scene mining result in a pre-established scene knowledge base according to the target scene element; wherein the scene knowledge base includes a plurality of scene codes, each scene code includes a plurality of scene element codes; and the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image.

[0058] In a preferred solution, after receiving the mining instruction from the user, the method further includes:

[0059] Determining whether the mining instruction is a first-round instruction;

[0060] If the mining instruction is not a first-round instruction, the mining instruction is used as a current-round instruction to obtain the scene mining results matched by the previous-round instruction;

[0061] Based on the scene mining results of the previous round of instruction matching, the scene mining results of this round of instruction matching are determined in a pre-established scene knowledge base.

[0062] In order to understand the user's intention more clearly, the test platform can conduct multiple interactive questions and answers with the user to perform refined scenario mining. Specifically, after receiving the user's mining instruction, the test platform can determine whether the mining instruction is the first round of instructions. The first round of instructions can be the mining instruction initiated by the user for the first time during the continuation of a mining task. Specifically, the test platform can determine whether two adjacent mining instructions belong to the same mining task based on the interval between the mining instructions. For example, if the interval between two adjacent mining instructions is greater than the preset time threshold, it is determined that the two adjacent mining instructions do not belong to the same mining task. If the interval between two adjacent mining instructions is less than or equal to the preset time threshold, it is determined that the two adjacent mining instructions belong to the same mining task.

[0063] If it is detected that the mining instruction is the first round instruction of the current mining task, the test platform can directly perform scene mining according to steps S250-S260. If the mining instruction is not the first round instruction, the test platform can use the mining instruction as the current round instruction, obtain the scene mining results matched by the previous round instruction, and further perform scene mining in the pre-established scene knowledge base on the scene mining results matched by the previous round instruction, and determine the scene mining results matched by the current round instruction. The test platform can perform scene mining in the scene knowledge base based on RAG (Retrieval-augmented Generation).

[0064] Specifically, the test platform can search for the scene codes that match the current round of instructions in the scene knowledge base, associate the scene codes that match the current round of instructions with the scene mining results of the previous round of instructions, and generate the scene mining results that match the current round of instructions. For example, the scene mining results of the previous round of instructions are driving scenes with more than 12 lanes, and the scene codes that match the current round of instructions are scene codes for driving scenes with bicycles. The test platform can infer that the user needs to find a driving scene at an intersection that includes both motor vehicle lanes and non-motor vehicle lanes based on the scene mining results of the previous round of instructions and the scene codes that match the current round of instructions, and thus use the scene codes of the above driving scenes as the scene mining results of the current round of instructions.

[0065] The technical solution of the embodiment of the present invention, if a mining instruction from a user is received, then based on a language processing model, at least one target scene element is extracted from the mining instruction; based on the target scene element, the scene mining result is determined in a pre-established scene knowledge base; wherein the scene knowledge base includes multiple scene codes, each scene code includes multiple scene element codes; the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image. This technical solution solves the problem of poor flexibility in autonomous driving scene mining, and based on the fusion of multimodal scene data and question-answer interaction, effectively improves the recognition ability of new and complex scenes, reduces dependence on manual and professional knowledge, and enhances the intelligence and consistency of the mining process.

[0066] Embodiment 3

[0067] Figure 3 This is a schematic diagram of the structure of an excavation device for an autonomous driving scenario provided by the third embodiment of the present invention. Figure 3 As shown, the device comprises:

[0068] A scene element extraction module 310 is configured to extract at least one target scene element from the mining instruction based on a language processing model upon receiving a mining instruction from the user;

[0069] A mining result determination module 320 is used to determine a scene mining result in a pre-established scene knowledge base according to the target scene elements;

[0070] The scene knowledge base includes a plurality of scene codes, each of which includes a plurality of scene element codes; the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image.

[0071] In this solution, optionally, the device further includes a knowledge base establishment module, and the knowledge base establishment module includes:

[0072] An image perception unit, used to acquire a driving scene image, perform visual perception processing on the driving scene image, and output a visual perception result;

[0073] A text generation unit, configured to generate a scene description text matched with the driving scene image according to the visual perception result;

[0074] A code generation unit, configured to generate a scene code according to the driving scene image and a scene description text matched with the driving scene image;

[0075] The knowledge base building unit is used to build a scene knowledge base according to the scene codes matched by each driving scene image in the scene sample set.

[0076] In a feasible solution, the visual perception processing includes at least one of target detection, target segmentation and target recognition; the targets are people and objects in the driving scene.

[0077] In this embodiment, the code generation unit is specifically used to:

[0078] A scene code is generated according to the driving scene image and the scene description text matched with the driving scene image based on a visual language fusion model.

[0079] Based on the above scheme, the visual language fusion model includes an image encoder, a text encoder and a feature fuser;

[0080] The code generation unit is specifically used for:

[0081] Inputting the driving scene image into the image encoder to obtain image features, and inputting the scene description text matched with the driving scene image into the text encoder to obtain text features;

[0082] Inputting the image features and the text features into the feature fusion device to obtain multiple scene element codes;

[0083] Multiple scene element codes are combined to obtain a scene code.

[0084] In a preferred embodiment, the device further comprises:

[0085] A round determination module, configured to determine whether the mining instruction is a first round instruction after receiving the mining instruction from the user;

[0086] A mining result acquisition module, used to, if the mining instruction is not a first-round instruction, use the mining instruction as a current-round instruction to acquire the scene mining results matched by the previous-round instruction;

[0087] The current round result determination module is used to determine the scene mining results of the current round of instruction matching in a pre-established scene knowledge base based on the scene mining results of the previous round of instruction matching.

[0088] The mining device for autonomous driving scenes provided in an embodiment of the present invention can execute the mining method for autonomous driving scenes provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0089] Embodiment 4

[0090] Figure 4 A schematic diagram of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0091] like Figure 4 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0092] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0093] Processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. Processor 411 executes the various methods and processes described above, such as the mining method of autonomous driving scenarios.

[0094] In some embodiments, the mining method of the autonomous driving scene may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the mining method of the autonomous driving scene described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to execute the mining method of the autonomous driving scene in any other appropriate manner (e.g., by means of firmware).

[0095] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] The computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable autonomous driving scenario mining device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0098] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0099] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0100] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0101] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0102] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for mining autonomous driving scenarios, characterized in that: The method comprises: If a mining instruction from a user is received, extracting at least one target scene element from the mining instruction based on a language processing model; Determining a scene mining result in a pre-established scene knowledge base according to the target scene elements; The scene knowledge base includes a plurality of scene codes, each of which includes a plurality of scene element codes; the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image.

2. The method according to claim 1, characterized in that The process of establishing the scenario knowledge base includes: Acquire a driving scene image, perform visual perception processing on the driving scene image, and output a visual perception result; Generating a scene description text matched with the driving scene image according to the visual perception result; generating a scene code according to the driving scene image and a scene description text matched with the driving scene image; A scene knowledge base is established based on the scene coding matched with each driving scene image in the scene sample set.

3. The method according to claim 2, characterized in that The visual perception processing includes at least one of target detection, target segmentation and target recognition; the targets are people and objects in the driving scene.

4. The method according to claim 2, characterized in that: The generating a scene code according to the driving scene image and the scene description text matched with the driving scene image comprises: A scene code is generated according to the driving scene image and the scene description text matched with the driving scene image based on a visual language fusion model.

5. The method according to claim 4, characterized in that The visual language fusion model includes an image encoder, a text encoder and a feature fuser; The step of generating a scene code based on the driving scene image and the scene description text matched with the driving scene image and based on a visual language fusion model includes: Inputting the driving scene image into the image encoder to obtain image features, and inputting the scene description text matched with the driving scene image into the text encoder to obtain text features; Inputting the image features and the text features into the feature fusion device to obtain multiple scene element codes; Multiple scene element codes are combined to obtain a scene code.

6. The method according to claim 1, characterized in that After receiving the mining instruction from the user, the method further includes: Determining whether the mining instruction is a first-round instruction; If the mining instruction is not a first-round instruction, the mining instruction is used as a current-round instruction to obtain the scene mining results matched by the previous-round instruction; Based on the scene mining results of the previous round of instruction matching, the scene mining results of this round of instruction matching are determined in a pre-established scene knowledge base.

7. An excavation device for an autonomous driving scenario, characterized in that: The device comprises: A scene element extraction module, configured to extract at least one target scene element from the mining instruction based on a language processing model upon receiving a mining instruction from the user; A mining result determination module, used to determine the scene mining result in a pre-established scene knowledge base according to the target scene elements; The scene knowledge base includes a plurality of scene codes, each of which includes a plurality of scene element codes; the scene code is determined based on a driving scene image and a scene description text matched with the driving scene image.

8. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the mining method of the autonomous driving scenario described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for mining the autonomous driving scenario according to any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method for mining an autonomous driving scenario according to any one of claims 1 to 6.

Images