Vehicle key information extraction method, device, system and electronic equipment

Abstract

The application discloses a vehicle key information extraction method, device and system and electronic equipment, and relates to the technical field of vehicles. The method comprises the following steps: acquiring video data and CAN data of a vehicle; for each video frame in the video data, if a time difference between the video frame and a starting time of the video data is greater than a preset time, a first reference frame is selected from the video data according to a first time corresponding to the video frame, and a pixel difference value between the first reference frame and the video frame is obtained; a key frame is selected from the video data according to the pixel difference value and the CAN data; and a key information extraction result is generated according to the key frame. Thus, the vehicle video data, including a vehicle video without audio text information, can be subjected to key information extraction.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and in particular to a method, apparatus, system, and electronic device for extracting key vehicle information. Background Technology

[0002] One of the core aspects of intelligent driving is the extraction of key information from vehicle video data. In related technologies, key information extraction is achieved by analyzing the motion trajectory and scale changes of spatiotemporal slices. However, this method is only applicable to static situations where neither the vehicle nor the background changes, and it is difficult to adapt to the vehicle environment. Alternatively, video frames can be extracted using methods such as shot segmentation and clustering, followed by keyword-based visual content summarization, and summarization of the original audio and text to obtain the final summary of the video. However, this method is only applicable to videos with audio and text information and is difficult to process videos without audio and text information, such as vehicle driving records and surveillance videos. Summary of the Invention

[0003] This invention aims to at least partially address one of the technical problems in related technologies. Therefore, the first objective of this invention is to provide a method for extracting key vehicle information from vehicle video data, including video without audio or text information.

[0004] The second objective of this invention is to provide an electronic device.

[0005] The third objective of this invention is to provide a vehicle key information extraction device.

[0006] The fourth objective of this invention is to provide a vehicle key information extraction system.

[0007] To avoid failing to achieve the above objectives, a first aspect of the present invention proposes a method for extracting key vehicle information. The method includes: acquiring video data and CAN data of a vehicle; for each video frame in the video data whose time difference with the start time of the video data is greater than a preset time, selecting a first reference frame from the video data according to a first time corresponding to the video frame, and obtaining a pixel difference value between the first reference frame and the video frame; selecting a key frame from the video data according to the pixel difference value and the CAN data; and generating a key information extraction result based on the key frame.

[0008] In addition, the vehicle key information extraction method according to embodiments of the present invention may also have the following additional technical features: According to an embodiment of the present invention, the step of selecting a first reference frame from the video data based on a first moment corresponding to the video frame and obtaining a pixel difference value between the first reference frame and the video frame includes: obtaining a historical frame from the video data based on the first moment, wherein a second moment corresponding to the historical frame is earlier than the first moment, and the difference between the second moment and the first moment is within a preset time range, wherein the maximum value of the preset time range is less than or equal to the preset time; subtracting the pixel value of the corresponding pixel in the historical frame from the pixel value of each pixel in the video frame to obtain a pixel difference value, and averaging the absolute values ​​of the pixel differences to obtain the pixel difference value corresponding to the video frame.

[0009] According to an embodiment of the present invention, the step of selecting keyframes from video data based on the pixel difference value and the CAN data includes: selecting a plurality of second reference frames from the video data based on the pixel difference value and the CAN data; obtaining a frame set based on the plurality of second reference frames; obtaining a third time point corresponding to each second reference frame in the frame set, and obtaining the mileage traveled by the vehicle at each third time point based on the CAN data; arranging the plurality of third time points in a preset order to obtain a first time point sequence; obtaining the mileage difference between the mileage traveled between any two adjacent time points in the first time point sequence; for each mileage difference, comparing the mileage difference with a preset difference threshold; when the mileage difference is less than the preset difference threshold, removing the two second reference frames corresponding to the mileage difference from the frame set, averaging the two first time points corresponding to the mileage difference to obtain an average time point, and adding the frame in the video data corresponding to the average time point to the frame set; and using the frames in the frame set and the first frame of the video data as the keyframes.

[0010] According to one embodiment of the present invention, a plurality of second reference frames include a third reference frame selected from the video data based on the pixel difference values ​​and a fourth reference frame selected from the video data based on the CAN data. The method for selecting the third reference frame includes: obtaining a first difference value among the pixel difference values ​​that is greater than a preset difference threshold; sorting all the first difference values ​​in descending order, and taking the first difference value that ranks first and accounts for N1% of the total number of the first difference values ​​as a second difference value, wherein N1 is greater than 0; and selecting the third reference frame from the video data based on the second difference value.

[0011] According to an embodiment of the present invention, the method for selecting the fourth reference frame includes: obtaining the first turning angle value of the vehicle at different times based on the CAN data; taking the first turning angle value that is greater than a preset turning angle threshold as the second turning angle value; sorting all the second turning angle values ​​in descending order, and taking the second turning angle value that ranks first and accounts for N2% of the total number of the second turning angle values ​​as the third turning angle value, wherein N2 is greater than 0; and selecting the fourth reference frame from the video data based on the third turning angle value.

[0012] According to one embodiment of the present invention, the step of generating key information extraction results based on the key frame includes: obtaining the vehicle turning angle value and the vehicle mileage corresponding to the key frame in CAN data; inputting the key frame into a graph-to-text model to obtain a text description of the key frame; and inputting the text description, the vehicle turning angle value, and the vehicle mileage into a language model to generate the key information extraction results.

[0013] According to one embodiment of the present invention, before inputting the text description, the vehicle turning angle value, and the vehicle's mileage into a language model to generate the key information extraction result, the method further includes: obtaining traffic regulation text; and generating prompt words and / or reference information for the language model based on the traffic regulation text.

[0014] To achieve the above objectives, a second aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, it implements the above-described method for extracting key vehicle information.

[0015] To achieve the above objectives, a third aspect of the present invention provides a vehicle key information extraction device, the device comprising: an acquisition module, configured to acquire video data and CAN data of a vehicle, and for each video frame in the video data whose time difference with the start time of the video data is greater than a preset time, to select a reference frame from the video data according to a first time corresponding to the video frame, and obtain a pixel difference value between the reference frame and the video frame; a selection module, configured to select a key frame from the video data according to the pixel difference value and the CAN data; and a generation module, configured to generate a key information extraction result based on the key frame.

[0016] To achieve the above objectives, a fourth aspect of the present invention provides a vehicle key information extraction system, including the vehicle key information extraction device described above.

[0017] According to embodiments of the present invention, a method, apparatus, system, and electronic device for extracting key vehicle information acquire vehicle video data and CAN data; for each video frame in the video data whose time difference with the start time of the video data is greater than a preset time, a first reference frame is selected from the video data according to the first time corresponding to the video frame, and the pixel difference value between the first reference frame and the video frame is obtained; a key frame is selected from the video data based on the pixel difference value and the CAN data; and a key information extraction result is generated based on the key frame. Thus, key information can be extracted from vehicle video data, including videos of vehicles without audio or text information.

[0018] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] Figure 1 This is a flowchart of the vehicle key information extraction method according to an embodiment of the present invention; Figure 2 This is a flowchart of a method for extracting key vehicle information according to an embodiment of the present invention; Figure 3 This is a structural block diagram of an electronic device according to an embodiment of the present invention; Figure 4 This is a structural block diagram of the vehicle key information extraction device according to an embodiment of the present invention; Figure 5 This is a structural block diagram of the vehicle key information extraction system according to an embodiment of the present invention. Detailed Implementation

[0020] The following description, with reference to the accompanying drawings, outlines a method, apparatus, system, and electronic device for extracting key vehicle information according to embodiments of the present invention. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. The embodiments described with reference to the accompanying drawings are exemplary and should not be construed as limiting the present invention.

[0021] Figure 1 This is a flowchart of a method for extracting key vehicle information according to an embodiment of the present invention.

[0022] like Figure 1 As shown, the method for extracting key vehicle information includes: S11 acquires video and CAN data from the vehicle.

[0023] See Figure 2 First, the vehicle data is acquired, which includes video data and CAN data. The video data is the data to be extracted, which can be video clips without audio or text information, such as vehicle recordings or surveillance videos.

[0024] S12, for each video frame in the video data whose time difference with the start time of the video data is greater than a preset time, a first reference frame is selected from the video data according to the first time corresponding to the video frame, and the pixel difference value between the first reference frame and the video frame is obtained.

[0025] Specifically, see Figure 2 After obtaining the driving data, keyframe extraction is performed on the video data in the driving data.

[0026] To achieve keyframe extraction, the inter-frame difference is first quantized using a differential method. Specifically, the above-mentioned selection of a first reference frame from the video data based on the first moment corresponding to the video frame, and obtaining the pixel difference value between the first reference frame and the video frame, includes: obtaining a historical frame from the video data based on the first moment, wherein the second moment corresponding to the historical frame is earlier than the first moment, and the difference between the second moment and the first moment is within a preset time range, the maximum value of the preset time range being less than or equal to the preset time; subtracting the pixel value of the corresponding pixel in the historical frame from the pixel value of each pixel in the video frame to obtain the pixel difference value, and averaging the absolute values ​​of the pixel differences to obtain the pixel difference value corresponding to the video frame.

[0027] The time value within the aforementioned preset time range must be neither too small nor too large. If it is too small, the difference will be too small; if it is too large, the difference will be too large, making it difficult to generate a peak value, which is not conducive to subsequent screening. According to actual testing, it can be 100~500ms, that is, the aforementioned preset time must be greater than or equal to 500ms. Assuming it is 501ms, then each video frame in the aforementioned video data whose time difference with the start time of the video data is greater than the preset time refers to each frame in the video data after 501ms. That is, for each frame, it is one frame within 100~500ms before it.

[0028] S13, select keyframes from video data based on pixel difference values ​​and CAN data.

[0029] Specifically, see Figure 2 The above-mentioned selection of key frames from video data based on pixel difference values ​​and CAN data includes: selecting multiple second reference frames from video data based on pixel difference values ​​and CAN data.

[0030] The aforementioned plurality of second reference frames include a third reference frame selected from the video data based on pixel difference values ​​and a fourth reference frame selected from the video data based on CAN data. That is, at least one third reference frame needs to be selected from the video data based on pixel difference values, and at least one fourth reference frame needs to be selected from the video data based on CAN data, and then both the selected third and fourth reference frames are used as second reference frames.

[0031] The third reference frame is Figure 2The image shows the search results for a local maximum search based on pixel difference values, with the fourth reference frame being... Figure 2 The search results shown are for a local maximum search based on historical angle values ​​in CAN data.

[0032] Therefore, it is possible to achieve Figure 2 The local maximum search performed based on pixel difference values ​​and CAN data is shown.

[0033] After obtaining multiple second reference frames through local maximum search, execution can proceed. Figure 2 The method shown filters frames with similar mileage based on historical mileage in CAN data. This historical mileage can be the vehicle's mileage already traveled. Specifically, a frame set can be obtained based on multiple second reference frames; a third time point corresponding to each second reference frame in the frame set can be obtained, and the mileage already traveled by the vehicle at each third time point can be obtained based on CAN data; multiple third time points can be arranged in a preset order to obtain a first time point sequence; the mileage difference between the mileage already traveled at any two adjacent time points in the first time point sequence can be obtained; for each mileage difference, the mileage difference is compared with a preset difference threshold. When the mileage difference is less than the preset difference threshold, the two second reference frames corresponding to the mileage difference are removed from the frame set, and the two first time points corresponding to the mileage difference are averaged to obtain an average time point. The frame in the video data corresponding to the average time point is added to the frame set; the frames in the frame set and the first frame of the video data are used as keyframes.

[0034] Among them, the frames in the video data that correspond to the average time can be obtained by acquiring the time corresponding to each frame in the video data, and finding the frames whose times corresponding to each frame in the video data are closest to the average time.

[0035] Therefore, it can avoid the impact on the extraction of key vehicle information in situations where the vehicle is stationary, such as when the vehicle is parked, resulting in a large amount of video data of the same scene.

[0036] The method for selecting the third reference frame includes: acquiring a first difference value among the pixel difference values ​​that is greater than a preset difference threshold; sorting all the first difference values ​​in descending order, and taking the first difference values ​​that rank first and account for N1% of the total number of first difference values ​​as the second difference value, where N1 is greater than 0; and selecting a third reference frame from the video data based on the second difference value. N1 is a pre-tuned value, for example, it can be 1.

[0037] The above method of selecting a third reference frame from the video data based on the second difference value can obtain the video frame corresponding to each second difference value and use that video frame as the first initial frame.

[0038] Next, obtain the time interval between any two first initial frames. If the duration of this time interval is less than the first preset time threshold (which needs to be pre-adjusted, for example, 3 seconds), change the identities of the two first initial frames and the first initial frames corresponding to the times within this time interval from first initial frames to first discard frames. Then, select a time interval from this time interval, for example, take the average of the times corresponding to the first discard frames, and take the frame in the video data whose corresponding time is closest to the time corresponding to this average as the new first initial frame.

[0039] After performing the above processing on any two first initial frames, each first initial frame is used as a third reference frame.

[0040] The method for selecting the aforementioned fourth reference frame includes: obtaining the vehicle's first steering angle value at different times based on CAN data. This first steering angle value is... Figure 2 The historical turning angle value shown can be information obtained using the vehicle's odometer.

[0041] Furthermore, the first corner value that is greater than a preset corner threshold is taken as the second corner value; all the second corner values ​​are sorted in descending order, and the top-ranked second corner value, which accounts for N2% of the total number of second corner values, is taken as the third corner value, where N2 is greater than 0; a fourth reference frame is selected from the video data based on the third corner value. The above N2 is a pre-tuned value, for example, it can be 1.

[0042] The above-mentioned selection of the fourth reference frame from the video data based on the third turning angle value can obtain the time when the vehicle acquires the third turning angle value for each third turning angle value, and obtain the corresponding time of each frame in the video data. The frame with the closest corresponding time in each frame of the video data to the time when the vehicle acquires the third turning angle value is taken as a second initial frame.

[0043] Next, obtain the time interval between any two times corresponding to the second initial frames. If the duration of this time interval is less than the second preset time threshold (which needs to be pre-adjusted, for example, 3 seconds), convert the identities of the two second initial frames and the second initial frames corresponding to the times within this time interval from second initial frames to second discard frames. Then, select a time from this time interval, for example, take the average of the times corresponding to the second discard frames, and take the frame in the video data whose corresponding time is closest to the time corresponding to this average as the new second initial frame.

[0044] After performing the above processing on any two second initial frames, each second initial frame is used as a fourth reference frame.

[0045] It should be noted that the time corresponding to the frame mentioned above refers to the time when the device that captured the video data on the vehicle captured the frame. For example, the first time corresponding to the video frame mentioned above refers to the time when the device that captured the video data on the vehicle captured the video frame.

[0046] S14, Generate key information extraction results based on key frames.

[0047] Specifically, see Figure 2 After filtering frames with similar mileage, large-scale model inference can be performed to generate key information extraction results based on key frames. The vehicle can then use these key information extraction results to assess the situation and achieve intelligent driving based on the assessment results.

[0048] The above-mentioned key information extraction results generated based on key frames include: obtaining the vehicle turning angle value and vehicle mileage corresponding to the key frame in the CAN data; inputting the key frame into the graph-to-text big data model to obtain the text description of the key frame; and inputting the text description, vehicle turning angle value and vehicle mileage into the language big data model to generate key information extraction results.

[0049] Before inputting the text description, vehicle turning angle value, and vehicle mileage into the language big model to generate key information extraction results, the method also includes: obtaining traffic regulation text; generating prompt words and / or reference information for the language big model based on the traffic regulation text.

[0050] The vehicle angle value and vehicle mileage corresponding to the key frame in the above CAN data refer to the time when the key frame is obtained, the time when each vehicle angle value and vehicle mileage in the CAN data is obtained, and the vehicle angle value and vehicle mileage at the time when the key frame is obtained are used as the vehicle angle value and vehicle mileage corresponding to the key frame.

[0051] The above-mentioned large-scale image-to-text model can use open-source large-scale models, such as JoyCaption (an open-source visual language model specifically for image description) and Florence (an open-source visual language model specifically for image description).

[0052] See Figure 2 To achieve large-scale model inference, open-source graph-to-text models such as JoyCaption and Florence are first used to extract text descriptions from all keyframes. These text descriptions are... Figure 2The image-based inference text is shown. Before use, prompts need to be set for the large model, such as whether to focus on people and vehicles, whether to identify weather, whether to identify traffic congestion, and whether to identify highways or urban areas, to achieve prompt guidance. The input of this type of large model consists of two parts: an image and text. The image is the image to be analyzed, and the text can describe the semantic information that needs to be focused on.

[0053] The aforementioned large language model can use open-source large models, such as DeepSeek (a large model primarily focused on text understanding) and LlaMA (a large model primarily focused on text understanding).

[0054] See Figure 2 After obtaining the inference text, open-source language models such as DeepSeek and LLaMA can be used to summarize all text descriptions (including corresponding timestamps, steering angles, and mileage information), i.e., execute... Figure 2 The temporal analysis of the text shown yields descriptive text and keywords for the video clips. Before use, the large-scale model needs to be cued, such as focusing on temporal changes, analyzing vehicle dynamics, whether it has passed through tunnels or bridges, and identifying any potential violations or dangerous behaviors. The cues used can be generated based on pre-acquired traffic regulations. Relevant traffic regulations can be searched and sent to the language model for learning, saving the DST (Dialogue State Tracking). This DST serves as the aforementioned reference information and is called upon in each subsequent processing iteration.

[0055] In summary, the vehicle key information extraction method of this invention acquires vehicle video data and CAN data; for each video frame whose time difference with the start time of the video data is greater than a preset time, a first reference frame is selected from the video data according to the first time corresponding to the video frame, and the pixel difference value between the first reference frame and the video frame is obtained; key frames are selected from the video data according to the pixel difference value and CAN data; and key information extraction results are generated based on the key frames. Therefore, key information can be extracted from vehicle videos without audio or text information. Furthermore, the use of "difference method + local maximum search" is suitable for driving scenarios, runs quickly, and makes the method easy to implement; combining corner information and odometer information from CAN data can capture frames when the vehicle trajectory changes, making it suitable for key frame extraction of driving segments while reducing duplicate frames; combining graph-based text models and large language models, and incorporating traffic-related prompts or reference knowledge, makes the extracted results easier to apply to intelligent driving tasks.

[0056] Furthermore, the present invention proposes an electronic device.

[0057] Figure 3This is a structural block diagram of an electronic device according to an embodiment of the present invention.

[0058] like Figure 3 As shown, the electronic device 500 includes a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, for example, via a bus 502. Optionally, the electronic device 500 may also include a transceiver 504. It should be noted that in practical applications, the transceiver 504 is not limited to one type, and the structure of this electronic device 500 does not constitute a limitation on the embodiments of the present invention.

[0059] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 501 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0060] Bus 502 may include a pathway for transmitting information between the aforementioned components. Bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 502 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0061] The memory 503 stores a computer program corresponding to the vehicle key information extraction method of the above embodiments of the present invention. This computer program is controlled and executed by the processor 501. The processor 501 executes the computer program stored in the memory 503 to implement the content shown in the aforementioned method embodiments.

[0062] in, Figure 3 The electronic device 500 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0063] The electronic device of this invention, by implementing the vehicle key information extraction method of the above embodiments, can extract key information from video of a vehicle without audio text information.

[0064] Furthermore, the present invention proposes a device for extracting key vehicle information.

[0065] Figure 4 This is a structural block diagram of the vehicle key information extraction device according to an embodiment of the present invention.

[0066] like Figure 4 As shown, the vehicle key information extraction device 100 includes: an acquisition module 101, used to acquire video data and CAN data of the vehicle, and for each video frame in the video data whose time difference with the start time of the video data is greater than a preset time, to select a reference frame from the video data according to the first moment corresponding to the video frame, and obtain the pixel difference value between the reference frame and the video frame; a selection module 102, used to select key frames from the video data according to the pixel difference value and CAN data; and a generation module 103, used to generate key information extraction results according to the key frames.

[0067] It should be noted that other specific embodiments of the vehicle key information extraction device of the present invention can refer to the vehicle key information extraction method of the above embodiments.

[0068] The vehicle key information extraction device of this invention can extract key information from video of a vehicle without audio or text information.

[0069] Furthermore, this invention proposes a vehicle key information extraction system.

[0070] Figure 5 This is a structural block diagram of the vehicle key information extraction system according to an embodiment of the present invention.

[0071] like Figure 5 As shown, the vehicle key information extraction system 10 includes the aforementioned vehicle key information extraction device 100.

[0072] The vehicle key information extraction system of this invention can extract key information from vehicle videos without audio text information.

[0073] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein can be considered as a ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0074] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0075] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0076] In the description of this specification, the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and should not be construed as limiting the present invention.

[0077] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0078] In this specification, unless otherwise stated, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly defined. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0079] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0080] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for extracting key vehicle information, characterized in that, The method includes: Acquire video and CAN data from the vehicle; For each video frame in the video data whose time difference with the start time of the video data is greater than a preset time, a first reference frame is selected from the video data according to the first time corresponding to the video frame, and the pixel difference value between the first reference frame and the video frame is obtained. Keyframes are selected from the video data based on the pixel difference values ​​and the CAN data; Key information extraction results are generated based on the keyframes.

2. The method for extracting key vehicle information according to claim 1, characterized in that, The step of selecting a first reference frame from the video data based on the first moment corresponding to the video frame, and obtaining the pixel difference value between the first reference frame and the video frame, includes: Based on the first moment, a historical frame is obtained from the video data, wherein the second moment corresponding to the historical frame is earlier than the first moment, and the difference between the second moment and the first moment is within a preset time range, wherein the maximum value of the preset time range is less than or equal to the preset time. The pixel difference is obtained by subtracting the pixel value of the corresponding pixel in the historical frame from the pixel value of each pixel in the video frame, and then averaging the absolute values ​​of the pixel differences to obtain the pixel difference value corresponding to the video frame.

3. The method for extracting key vehicle information according to claim 1, characterized in that, The step of selecting keyframes from the video data based on the pixel difference value and the CAN data includes: Multiple second reference frames are selected from the video data based on the pixel difference value and the CAN data, and a frame set is obtained based on the multiple second reference frames; The third time point corresponding to each second reference frame in the frame set is obtained, and the mileage driven by the vehicle at each third time point is obtained based on the CAN data; Arrange the multiple third moments in a preset order to obtain the first moment sequence; The mileage difference between any two adjacent times in the first time sequence is obtained. For each mileage difference, the mileage difference is compared with a preset difference threshold. When the mileage difference is less than the preset difference threshold, the two second reference frames corresponding to the mileage difference are removed from the frame set, and the two first moments corresponding to the mileage difference are averaged to obtain the average moment. The frames in the video data corresponding to the average moment are added to the frame set. The frames in the frame set and the first frame of the video data are used as the keyframes.

4. The method for extracting key vehicle information according to claim 3, characterized in that, The plurality of second reference frames includes a third reference frame selected from the video data based on the pixel difference value and a fourth reference frame selected from the video data based on the CAN data, wherein the method for selecting the third reference frame includes: Obtain the first difference value among the pixel difference values ​​that is greater than a preset difference threshold; Sort all the first difference values ​​in descending order, and take the first difference values ​​that are at the top of the sort and account for N1% of the total number of the first difference values ​​as the second difference values, where N1 is greater than 0; The third reference frame is selected from the video data based on the second difference value.

5. The method for extracting key vehicle information according to claim 4, characterized in that, The method for selecting the fourth reference frame includes: The first turning angle value of the vehicle at different times is obtained based on the CAN data; The first angle value that is greater than the preset angle threshold is taken as the second angle value; Sort all the second corner values ​​in descending order, and take the first second corner value, which accounts for N2% of the total number of second corner values, as the third corner value, where N2 is greater than 0; The fourth reference frame is selected from the video data based on the third corner value.

6. The method for extracting key vehicle information according to claim 1, characterized in that, The step of generating key information extraction results based on the keyframes includes: Obtain the vehicle turning angle value and vehicle mileage corresponding to the key frame from the CAN data; Input the keyframe into the graph-to-text model to obtain the text description of the keyframe; The text description, the vehicle's turning angle value, and the vehicle's mileage are input into the language model to generate the key information extraction results.

7. The method for extracting key vehicle information according to claim 6, characterized in that, Before inputting the text description, the vehicle turning angle value, and the vehicle's mileage into the language big data model to generate the key information extraction result, the method further includes: Obtain the text of traffic regulations; Based on the traffic regulations text, prompt words and / or reference information for the language model are generated.

8. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, it implements the vehicle key information extraction method according to any one of claims 1-7.

9. A vehicle key information extraction device, characterized in that, The device includes: The acquisition module is used to acquire video data and CAN data of the vehicle, and for each video frame in the video data whose time difference with the start time of the video data is greater than a preset time, a reference frame is selected from the video data according to the first time corresponding to the video frame to obtain the pixel difference value between the reference frame and the video frame. A selection module is used to select keyframes from the video data based on the pixel difference values ​​and the CAN data; The generation module is used to generate key information extraction results based on the key frames.

10. A vehicle key information extraction system, characterized in that, Includes the vehicle key information extraction device according to claim 9.

Images