Multi-scale driving scene understanding architecture, method and system

Abstract

The invention discloses a multi-scale driving scene understanding architecture, method and system. The architecture comprises a large language model, a global information encoder and a detail information encoder. The input data of the multi-scale driving scene understanding architecture comprises overall character description, detail character description and a driving scene visual image; the input of the large language model is overall character description and detail character description, the output of the large language model is global character features and detail character features, the input of the global information encoder and the input of the detail information encoder are driving scene visual images, the output of the global information encoder is global image features, and the output of the global information encoder is global image features. The output of the detail information encoder is the detail image features, and the multi-scale driving scene understanding architecture performs multi-scale driving scene understanding through the global character features, the global image features, the detail character features and the detail image features. According to the method, rich advanced semantic information in text description is fully utilized, and the comprehensiveness, integrity and accuracy of scene understanding are improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to a multi-scale driving scene understanding architecture, method and system thereof. Background Art

[0002] Driving scene understanding is a key technology in autonomous driving and advanced driving assistance systems (ADAS), which involves comprehensive perception and analysis of road environments, traffic participants, and potential dangers. With the development of deep learning technology, vision-based scene understanding has made significant progress. Among them, cross-modal semantic alignment is a technology that maps and matches information from different modal data (for example, images and text). In driving scene understanding, this technology can associate data information with semantic descriptions, build a bridge between images and text, and improve the accuracy and comprehensiveness of scene understanding.

[0003] However, most existing technical solutions combine basic technologies such as classification, detection and segmentation to process the content in driving scenes, and can only provide the most basic category information. Moreover, the technology is only single-modal, and the results cannot reflect the relationship between entities in the scene, nor do they provide a comprehensive understanding and analysis of the scene. Summary of the invention

[0004] The present application provides a multi-scale driving scene understanding architecture, method and system thereof, which solves at least one problem in the above-mentioned prior art.

[0005] The specific technical solutions are as follows:

[0006] In a first aspect, an embodiment of the present application provides a multi-scale driving scene understanding architecture, including: a large language model, a global information encoder, and a detail information encoder;

[0007] The input data of the multi-scale driving scene understanding architecture includes an overall text description, a detailed text description and a visual image of the driving scene; the input of the large language model is the overall text description and the detailed text description, the output of the large language model is global text features and detailed text features, the input of the global information encoder and the detailed information encoder are both the visual image of the driving scene, the output of the global information encoder is the global image feature, and the output of the detailed information encoder is the detailed image feature. The multi-scale driving scene understanding architecture performs multi-scale driving scene understanding through the global text features, global image features, the detailed text features and the detailed image features.

[0008] In some embodiments of the present application, both the global information encoder and the detail information encoder are based on the classic CNN model or the ViT visual encoder extended into a parallel multi-level structure.

[0009] In some embodiments of the present application, the global information encoder and the detail information encoder perform global information exchange, and the network intermediate features of the global information encoder are transferred and integrated into the corresponding network nodes of the detail information encoder.

[0010] In a second aspect, an embodiment of the present application provides a multi-scale driving scene understanding method, including:

[0011] Acquire a number of driving scene visual images of different working conditions, different times and different driving scene types, and perform natural language annotation of global dimension information and detailed image content on each of the driving scene visual images to obtain image triplet samples; wherein the image triplet samples are represented as {driving scene visual image, overall text description, detailed text description};

[0012] Constructing a multi-scale driving scene understanding architecture, the multi-scale driving scene understanding architecture comprising a large language model, a global information encoder and a detail information encoder, the input of the large language model is the overall text description and the detail text description, the output of the large language model is a global text feature and a detail text feature, the input of the global information encoder and the detail information encoder are both the driving scene visual image, the output of the global information encoder is a global image feature, and the output of the detail information encoder is a detail image feature;

[0013] The multi-scale driving scene understanding framework is trained using the image triplet samples to obtain the trained multi-scale driving scene understanding framework. The cross-modal semantic alignment of visual features and text features in the same scene is achieved through the trained multi-scale driving scene understanding framework to obtain global text features, global image features, detailed text features, and detailed image features after cross-modal semantic alignment, so as to perform multi-scale driving scene understanding based on the global text features, the global image features, the detailed text features, and the detailed image features.

[0014] In some embodiments of the present application, both the global information encoder and the detail information encoder are based on the classic CNN model or the ViT visual encoder extended into a parallel multi-level structure.

[0015] In some embodiments of the present application, the global information encoder and the detail information encoder perform global information exchange, and the network intermediate features of the global information encoder are transferred and integrated into the corresponding network nodes of the detail information encoder.

[0016] In some embodiments of the present application, the global information encoder and the detail information encoder are trained using a contrastive learning method.

[0017] In a third aspect, an embodiment of the present application provides a multi-scale driving scene understanding system, including:

[0018] A data construction module is used to obtain a number of driving scene visual images of different working conditions, different times and different driving scene types, and to perform natural language annotation of global dimension information and detailed image content on each of the driving scene visual images to obtain image triplet samples; wherein the image triplet samples are represented as {driving scene visual image, overall text description, detailed text description};

[0019] An architecture construction module is used to construct a multi-scale driving scene understanding architecture, wherein the multi-scale driving scene understanding architecture includes a large language model, a global information encoder, and a detail information encoder. The input of the large language model is the overall text description and the detail text description, and the output of the large language model is a global text feature and a detail text feature. The input of the global information encoder and the detail information encoder are both the driving scene visual image, the output of the global information encoder is a global image feature, and the output of the detail information encoder is a detail image feature. The module is used to train the multi-scale driving scene understanding architecture using the image triplet samples to obtain the trained multi-scale driving scene understanding architecture.

[0020] The multi-scale driving scene understanding module is used to achieve cross-modal semantic alignment of visual features and text features in the same scene through the trained multi-scale driving scene understanding architecture, and obtain global text features, global image features, detailed text features, and detailed image features after cross-modal semantic alignment, so as to perform multi-scale driving scene understanding based on the global text features, the global image features, the detailed text features, and the detailed image features.

[0021] In some embodiments of the present application, both the global information encoder and the detail information encoder are based on a classic CNN model or a ViT visual encoder extended into a parallel multi-level structure;

[0022] The global information encoder and the detail information encoder perform global information exchange, and transmit and fuse the network intermediate features of the global information encoder into the corresponding network nodes of the detail information encoder.

[0023] In some embodiments of the present application, the global information encoder and the detail information encoder are trained using a contrastive learning method.

[0024] In a fourth aspect, an embodiment of the present application provides a multi-scale driving scene understanding device, comprising: a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it executes the multi-scale driving scene understanding method as described in the second aspect.

[0025] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multi-scale driving scene understanding method as described in the second aspect is implemented.

[0026] The beneficial effects of the embodiments of the present application are as follows:

[0027] The application adopts a network architecture for multi-scale feature extraction, which can focus on the relationship between various subjects in the driving scene and conduct a multi-scale understanding of the driving scene from the whole to the part. It adopts a global-fine-grained local multi-modal semantic alignment method to achieve accurate matching of visual features and text descriptions of driving scenes. It also uses data from two modalities, visual images and text descriptions, and makes full use of the rich high-level semantic information in text descriptions to retain and understand the overall information and detailed information in the driving scene, thereby improving the comprehensiveness, completeness and accuracy of driving scene understanding. In addition, the application also collects and constructs special data for driving scenes, matches driving scene knowledge, and enhances the ability of the multi-scale driving scene understanding architecture to understand driving scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A schematic diagram of a multi-scale driving scene understanding architecture provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of a flow chart of a multi-scale driving scene understanding method provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the components of a multi-scale driving scene understanding system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The terms "including" and "having" in the embodiments of the present application and the accompanying drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0034] The mainstream scene understanding methods in the existing technology mostly extract information of different dimensions of the scene in isolation, such as location information and category information, and are all single-modal. They do not pay attention to the relationship between the various subjects in the scene and lack a multi-scale understanding of the scene from the whole to the part.

[0035] In view of the above problems in the prior art, the present application discloses a multi-scale driving scene understanding architecture. Figure 1As shown, the multi-scale driving scene understanding architecture 300 includes: a large language model 310, a global information encoder 320 and a detail information encoder 330. Specifically, the input data of the multi-scale driving scene understanding architecture includes an overall text description, a detail text description and a driving scene visual image. The global information encoder 320 and the detail information encoder 330 constitute a multi-scale scene parsing network architecture for multi-scale visual image information extraction. The inputs of the global information encoder 320 and the detail information encoder 330 are both driving scene visual images. The output of the global information encoder 320 is a global image feature, and the output of the detail information encoder 330 is a detail image feature. The output of the global information encoder 320 is a global image feature, and the output of the detail information encoder 330 is a detail image feature. The driving scene visual image is sent to the global information encoder 320 and the detail information encoder 330 respectively to obtain global image features and detail image features, wherein the global image features include but are not limited to global dimension information such as the type of the current driving scene and the weather, and the detail image features include but are not limited to fine-grained information such as the detailed content in the driving scene, the type of traffic participants and the behavior relationship of traffic participants. The multi-scale driving scene understanding architecture 300 performs fine text semantic information extraction based on a large language model 310. The input of the large language model 310 is the overall text description and the detailed text description, and the output of the large language model 310 is the global text features and the detailed text features. That is, the overall description and the detailed description corresponding to the driving scene are sent to the large language model 310 and the corresponding features are extracted. In this way, by using the large language model 310 to process the text information, the rich semantic information in the text is retained to the greatest extent. At the same time, the multi-scale driving scene understanding architecture 300 uses a contrastive learning method to achieve cross-modal semantic alignment, and realizes cross-modal semantic alignment of visual feature alignment and text feature in the same scene, so as to use the global text features, global image features, detailed text features, and detailed image features after cross-modal semantic alignment for multi-scale driving scene understanding, which can make full use of the rich high-level semantic information in the text description, improve the comprehensiveness, completeness and accuracy of scene understanding, and the multi-scale driving scene understanding architecture 300 simultaneously retains and understands the overall information (i.e., global text features, global image features) and detailed information (i.e., detailed text features, detailed image features), and can have a good understanding of the driving scene from the overall type and features to the local details, solving the problem of lack of multi-scale understanding of the scene from the whole to the local in the prior art.

[0036] In some embodiments, the multi-scale scene parsing network architecture is based on the classic CNN model or ViT visual encoder, and is expanded into a parallel multi-level structure, and the global information encoder 320 and the detail information encoder 330 encode and extract the overall information of the driving scene and the detail content information respectively. Furthermore, the global information encoder 320 and the detail information encoder 330 can interact with each other globally, and the network intermediate features of the global information encoder 320 are transferred and fused into the corresponding network nodes of the detail information encoder 330 to enhance the fine-grained content understanding ability of the detail information encoder 330.

[0037] The embodiment of the present application also discloses a multi-scale driving scene understanding method, the execution subject of which can be a device or equipment capable of realizing multi-scale driving scene understanding, which can be specifically deployed in an intelligent vehicle terminal or a cloud server. The multi-scale driving scene understanding method is described in detail below.

[0038] Figure 2 A multi-scale driving scene understanding method provided according to an embodiment of the present application is shown. Figure 2 As shown, the multi-scale driving scene understanding method includes the following steps:

[0039] Step S110: Acquire a number of driving scene visual images of different working conditions, different times and different driving scene types, and perform natural language annotation of global dimension information and detailed image content on each driving scene visual image to obtain image triplet samples.

[0040] The driving scene of the embodiments of the present application can be a structured road scene or an unstructured road scene such as a scenic spot. The technical personnel in this field can flexibly set the specific application scenarios according to actual needs, and this application does not make any specific limitations on this.

[0041] In an embodiment of the present application, for driving scene understanding, the multi-scale driving scene understanding method collects and constructs multi-modal dedicated data. Among them, the dedicated data includes several driving scene visual images of different working conditions, different times and different driving scene types, so as to train the constructed multi-scale driving scene understanding architecture through the dedicated data, and enhance the multi-scale driving scene understanding architecture's ability to understand the specific field of driving scenes. Furthermore, the "working conditions" mentioned above include but are not limited to regular traffic conditions, congested traffic conditions, severe weather conditions, etc., "time" includes but is not limited to daytime, evening, night, etc., and "driving scene types" include but are not limited to urban roads, highways, complex intersections, etc.

[0042] Specifically, the multi-scale driving scene understanding method widely collects driving scene visual images of different working conditions, different times and different scene types, and uses natural language to describe their global dimension information to obtain the corresponding overall text description, and also uses natural language to describe their detailed image content to obtain the corresponding detailed text description, for example, the "overall text description" is "a crossroads in clear weather", and the "detailed text description" is "the current vehicle is in the middle lane line, a car is parked on the right side of the road, and a pedestrian is crossing the road on the zebra crossing in front". Therefore, after natural language annotation, each driving scene visual image is expanded into a triple form including the driving scene visual image, the overall text description and the detailed text description, all the expanded data are used as image triple samples, and the image triple samples are represented as {driving scene visual image, overall text description, detailed text description}.

[0043] Step S120: construct a multi-scale driving scene understanding architecture.

[0044] In the embodiment of the present application, the multi-scale driving scene understanding architecture includes a large language model, a global information encoder and a detail information encoder, and its structure and principle are the same as those in the aforementioned multi-scale driving scene understanding architecture embodiment. For the sake of brief description, for matters not mentioned in the multi-scale driving scene understanding method embodiment, reference may be made to the corresponding content in the aforementioned multi-scale driving scene understanding architecture embodiment.

[0045] The constructed multi-scale driving scene understanding architecture can not only extract multi-scale visual image information, but also extract fine text semantic information, and realize cross-modal alignment of global-fine-grained local joint semantics, so that visual features can be accurately matched with text descriptions of driving scenes, thereby realizing multi-modal and multi-scale understanding of driving scenes from the whole to the local.

[0046] Step S130: training a multi-scale driving scene understanding framework using the image triplet samples to obtain a trained multi-scale driving scene understanding framework.

[0047] The multi-scale driving scene understanding architecture constructed in step S120 is trained using the image triplet samples obtained in step S110, so as to obtain a trained multi-scale driving scene understanding architecture.

[0048] In some embodiments, the large language model can select an open source trained model, or obtain a certain amount of corpus similar to the natural language description in step S110, and fine-tune the open source large language model using an autoregressive training method based on the obtained expectations. The global information encoder and the detail information encoder in the multi-scale scene parsing network architecture are trained using a contrastive learning method based on image triple samples to bring the features from different modalities representing the same semantic information as close as possible in the feature space, and the distance between features with different semantics as far as possible, that is, to ensure that the global features of the same driving scene (i.e., global text features, global image features) are as close as possible in the feature space, otherwise as far away as possible. Similarly, to ensure that the detail features of the same driving scene (i.e., detail text features, detail image features) are as close as possible in the feature space, otherwise as far away as possible, thereby achieving the purpose of semantic alignment.

[0049] Step S140: A cross-modal semantic alignment of visual features and text features in the same scene is achieved through the trained multi-scale driving scene understanding architecture, and global text features, global image features, detailed text features, and detailed image features after cross-modal semantic alignment are obtained, so as to perform multi-scale driving scene understanding based on the global text features, global image features, detailed text features, and detailed image features.

[0050] The trained multi-scale driving scene understanding architecture in the embodiment of the present application can achieve cross-modal semantic alignment of visual features and text features in the same scene. That is to say, the multi-scale driving scene understanding method not only has data in the modality of "visual image", but also introduces data in the modality of "text description", so that it can make full use of the rich high-level semantic information contained in the text description, improve the comprehensiveness, completeness and accuracy of driving scene understanding, and it uses multi-scale data of global features and detail features, which can have a better understanding of the driving scene from the overall to the local details.

[0051] Corresponding to the above method embodiment, the present application embodiment also provides a multi-scale driving scene understanding system, which is used to execute the multi-scale driving scene understanding method steps in the above embodiment. Figure 3 As shown, the multi-scale driving scene understanding system 200 includes a data construction module 210 , an architecture construction module 220 and a multi-scale driving scene understanding module 230 .

[0052] Specifically, the data construction module 210 is used to obtain a number of driving scene visual images of different working conditions, different times and different driving scene types, and to perform natural language annotation of global dimension information and detailed image content on each driving scene visual image to obtain image triplet samples. The image triplet samples are represented as {driving scene visual image, overall text description, detailed text description}.

[0053] The architecture construction module 220 is used to construct a multi-scale driving scene understanding architecture, wherein the multi-scale driving scene understanding architecture includes a large language model, a global information encoder and a detail information encoder. The input of the large language model is the overall text description and the detail text description, and the output of the large language model is the global text features and the detail text features. The input of the global information encoder and the detail information encoder are both driving scene visual images, the output of the global information encoder is the global image features, and the output of the detail information encoder is the detail image features. It is used to train the multi-scale driving scene understanding architecture using image triplet samples to obtain the trained multi-scale driving scene understanding architecture.

[0054] The multi-scale driving scene understanding module 230 is used to achieve cross-modal semantic alignment of visual features and text features in the same scene through the trained multi-scale driving scene understanding architecture, and obtain global text features, global image features, detailed text features, and detailed image features after cross-modal semantic alignment, so as to perform multi-scale driving scene understanding based on the global text features, global image features, detailed text features, and detailed image features.

[0055] Among them, the global information encoder and detail information encoder in the multi-scale driving scene understanding architecture constructed by the architecture construction module 220 are based on the classic CNN model or ViT visual encoder expanded into a parallel multi-level structure. Furthermore, the global information encoder and the detail information encoder interact with each other globally, and the network intermediate features of the global information encoder are transferred and fused into the corresponding network nodes of the detail information encoder. In addition, the global information encoder and detail information encoder in the multi-scale driving scene understanding architecture constructed by the architecture construction module 220 are trained using the network training method of contrastive learning.

[0056] It should be noted that the multi-scale driving scene understanding system provided in the embodiment of the present application is based on the same concept as the multi-scale driving scene understanding method embodiment of the present application, and the technical effect it brings is the same as the multi-scale driving scene understanding method embodiment of the present application. Therefore, the specific content can be found in the description in the multi-scale driving scene understanding method embodiment of the present application, and will not be repeated here.

[0057] The embodiment of the present application also provides a multi-scale driving scene understanding device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the multi-scale driving scene understanding method embodiment described in the above embodiment are implemented. Alternatively, when the processor executes the computer program, the functions of each module in the multi-scale driving scene understanding system embodiment described in the above embodiment are implemented.

[0058] In a specific embodiment, the computer program can be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the embodiment of the present invention. One or more modules can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the multi-scale driving scene understanding device. For example, the computer program can be divided into a data construction module, an architecture construction module, and a multi-scale driving scene understanding module, and the specific functions of each module are as follows:

[0059] The data construction module is used to obtain several driving scene visual images of different working conditions, different times and different driving scene types, and to annotate each driving scene visual image with global dimension information and detailed image content in natural language to obtain image triplet samples. The image triplet samples are represented as {driving scene visual image, overall text description, detailed text description}.

[0060] The architecture construction module is used to construct a multi-scale driving scene understanding architecture, wherein the multi-scale driving scene understanding architecture includes a large language model, a global information encoder and a detail information encoder. The input of the large language model is the overall text description and the detail text description, and the output of the large language model is the global text features and the detail text features. The input of the global information encoder and the detail information encoder are both driving scene visual images, the output of the global information encoder is the global image features, and the output of the detail information encoder is the detail image features. It is used to train the multi-scale driving scene understanding architecture using image triplet samples to obtain the trained multi-scale driving scene understanding architecture.

[0061] The multi-scale driving scene understanding module is used to achieve cross-modal semantic alignment of visual features and text features in the same scene through the trained multi-scale driving scene understanding architecture, and obtain global text features, global image features, detailed text features, and detailed image features after cross-modal semantic alignment, so as to perform multi-scale driving scene understanding based on global text features, global image features, detailed text features, and detailed image features.

[0062] The multi-scale driving scene understanding device may be a computing device such as a vehicle-mounted computer, a cloud management server, etc. Those skilled in the art may understand that the multi-scale driving scene understanding device may include but is not limited to a processor and a memory, and may also include more or fewer components, or a combination of certain components, or different components. For example, the multi-scale driving scene understanding device may also include input and output devices, network access devices, buses, etc.

[0063] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0064] The memory may be an internal storage unit of the multi-scale driving scene understanding device, for example, a hard disk or memory of the multi-scale driving scene understanding device. The memory may also be an external storage device of the multi-scale driving scene understanding device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital Card, referred to as SD card), a flash card (Flash Card), etc. equipped on the multi-scale driving scene understanding device. Further, the memory may also include both an internal storage unit of the multi-scale driving scene understanding device and an external storage device. The memory is used to store computer programs and other programs or data required by the multi-scale driving scene understanding device. The memory may also be used to temporarily store data that has been output or is to be output.

[0065] In the above embodiments, the description of each embodiment has different emphases. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0066] Those skilled in the art will appreciate that the modules and algorithm steps of the various embodiments described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but this implementation should not be considered to exceed the scope of the present invention.

[0067] In addition, the embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program can be executed by a processor to complete the multi-scale driving scene understanding method as described in the above embodiment. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), etc.

[0068] In summary, the present application discloses a multi-scale driving scene understanding architecture, method and system thereof, which adopts a network architecture for multi-scale feature extraction, can focus on the relationship between various subjects in the driving scene, and conduct multi-scale understanding of the driving scene from the whole to the part. It adopts a global-fine-grained local multi-modal semantic alignment method to achieve accurate matching of visual features and text descriptions of driving scenes, and uses data of two modalities, visual images and text descriptions, to make full use of the rich high-level semantic information in text descriptions, to retain and understand both the overall information and detailed information in the driving scene, and to improve the comprehensiveness, completeness and accuracy of driving scene understanding. In addition, the application also collects and constructs special data for driving scenes, matches driving scene knowledge, and enhances the ability of the multi-scale driving scene understanding architecture to understand driving scenes.

[0069] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0070] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-scale driving scene understanding architecture, characterized by: include: Large language model, global information encoder, and detail information encoder; The input data of the multi-scale driving scene understanding architecture includes an overall text description, a detailed text description and a visual image of the driving scene; the input of the large language model is the overall text description and the detailed text description, the output of the large language model is global text features and detailed text features, the input of the global information encoder and the detailed information encoder are both the visual image of the driving scene, the output of the global information encoder is the global image feature, and the output of the detailed information encoder is the detailed image feature. The multi-scale driving scene understanding architecture performs multi-scale driving scene understanding through the global text features, global image features, the detailed text features and the detailed image features.

2. The multi-scale driving scene understanding architecture according to claim 1, characterized in that: Both the global information encoder and the detail information encoder are based on the classic CNN model or the ViT visual encoder extended into a parallel multi-level structure.

3. The multi-scale driving scene understanding architecture according to claim 1, characterized in that: The global information encoder and the detail information encoder perform global information exchange, and transmit and fuse the network intermediate features of the global information encoder into the corresponding network nodes of the detail information encoder.

4. A multi-scale driving scene understanding method, characterized in that: include: Acquire a number of driving scene visual images of different working conditions, different times and different driving scene types, and perform natural language annotation of global dimension information and detailed image content on each of the driving scene visual images to obtain image triplet samples; wherein the image triplet samples are represented as {driving scene visual image, overall text description, detailed text description}; Constructing a multi-scale driving scene understanding architecture, the multi-scale driving scene understanding architecture comprising a large language model, a global information encoder and a detail information encoder, the input of the large language model is the overall text description and the detail text description, the output of the large language model is a global text feature and a detail text feature, the input of the global information encoder and the detail information encoder are both the driving scene visual image, the output of the global information encoder is a global image feature, and the output of the detail information encoder is a detail image feature; The multi-scale driving scene understanding framework is trained using the image triplet samples to obtain the trained multi-scale driving scene understanding framework. The cross-modal semantic alignment of visual features and text features in the same scene is achieved through the trained multi-scale driving scene understanding framework to obtain global text features, global image features, detailed text features, and detailed image features after cross-modal semantic alignment, so as to perform multi-scale driving scene understanding based on the global text features, the global image features, the detailed text features, and the detailed image features.

5. The multi-scale driving scene understanding method according to claim 4, characterized in that: Both the global information encoder and the detail information encoder are based on the classic CNN model or the ViT visual encoder extended into a parallel multi-level structure.

6. The multi-scale driving scene understanding method according to claim 4, characterized in that: The global information encoder and the detail information encoder perform global information exchange, and transmit and fuse the network intermediate features of the global information encoder into the corresponding network nodes of the detail information encoder.

7. The multi-scale driving scene understanding method according to claim 4, characterized in that: The global information encoder and the detail information encoder are trained using a contrastive learning method.

8. A multi-scale driving scene understanding system, characterized in that: include: A data construction module is used to obtain a number of driving scene visual images of different working conditions, different times and different driving scene types, and to perform natural language annotation of global dimension information and detailed image content on each of the driving scene visual images to obtain image triplet samples; wherein the image triplet samples are represented as {driving scene visual image, overall text description, detailed text description}; An architecture construction module is used to construct a multi-scale driving scene understanding architecture, wherein the multi-scale driving scene understanding architecture includes a large language model, a global information encoder, and a detail information encoder. The input of the large language model is the overall text description and the detail text description, and the output of the large language model is a global text feature and a detail text feature. The input of the global information encoder and the detail information encoder are both the driving scene visual image, the output of the global information encoder is a global image feature, and the output of the detail information encoder is a detail image feature. The module is used to train the multi-scale driving scene understanding architecture using the image triplet samples to obtain the trained multi-scale driving scene understanding architecture. The multi-scale driving scene understanding module is used to achieve cross-modal semantic alignment of visual features and text features in the same scene through the trained multi-scale driving scene understanding architecture, and obtain global text features, global image features, detailed text features, and detailed image features after cross-modal semantic alignment, so as to perform multi-scale driving scene understanding based on the global text features, the global image features, the detailed text features, and the detailed image features.

9. The multi-scale driving scene understanding system according to claim 8, characterized in that: The global information encoder and the detail information encoder are both based on the classic CNN model or the ViT visual encoder extended into a parallel multi-level structure; The global information encoder and the detail information encoder perform global information exchange, and transmit and fuse the network intermediate features of the global information encoder into the corresponding network nodes of the detail information encoder.

10. The multi-scale driving scene understanding system according to claim 8, characterized in that: The global information encoder and the detail information encoder are trained using a contrastive learning method.

Images