Methods and devices for obtaining traffic violation information

Abstract

This application provides a method and apparatus for obtaining traffic violation information. The method includes: obtaining a first input from a user, the first input being used to instruct a vehicle to obtain traffic violation information; in response to obtaining the first input, obtaining first data collected by a camera device outside the vehicle's cabin and second data collected by at least one radar; generating a traffic violation report text based on the first and second data, the traffic violation report text including multiple pieces of traffic violation information, the multiple pieces of traffic violation information including at least one of the following: violation time, violation location, license plate number, license plate color, and violation type; and controlling a prompting device of the vehicle to prompt the user with the traffic violation report text. The method for obtaining traffic violation information provided by this application can improve the accuracy of obtaining traffic violation information and improve the efficiency of reporting traffic violations, which is beneficial to improving the user experience.

Description

Technical Field

[0001] This application relates to the automotive field, and more specifically, to a method and apparatus for obtaining traffic violation information. Background Technology

[0002] Currently, vehicles have limited ability to judge traffic violations, resulting in poor accuracy of the obtained violation information. Furthermore, relying solely on manual detection and reporting of traffic violations is inefficient. Therefore, how to accurately and efficiently obtain violation information has become an urgent problem to be solved. Summary of the Invention

[0003] This application provides a method and apparatus for obtaining traffic violation information, which can improve the accuracy of obtaining traffic violation information and increase the efficiency of reporting traffic violations, thereby enhancing the user experience.

[0004] In a first aspect, a method for obtaining traffic violation information is provided. The method includes: obtaining a first input from a user, the first input being used to instruct a vehicle to obtain traffic violation information; in response to obtaining the first input, obtaining first data collected by a camera device outside the vehicle's cabin and second data collected by at least one radar; generating a traffic violation report text based on the first data and the second data, the traffic violation report text including multiple pieces of traffic violation information, the multiple pieces of traffic violation information including at least one of the following: violation time, violation location, license plate number, license plate color, and violation type; and controlling a prompting device of the vehicle to prompt the user with the traffic violation report text.

[0005] For example, the first input can be the user's voice, the text entered by the user through the display device, or the user's operation of clicking a button or control. The violation report text can be a colloquial description of the violation information, such as "The white vehicle with license plate ABC124 on the left illegally crossed the solid line while moving from the third lane to the second lane from the left, and was 20 meters away from this vehicle."

[0006] Based on the above technical solution, the vehicle can automatically determine a detailed description including traffic violation information through data obtained from multiple sensors after receiving user instructions. This avoids the process of users manually collecting traffic violation information, makes it easier for users to confirm traffic violation information, improves the accuracy of obtaining traffic violation information and increases the efficiency of reporting traffic violations, thereby improving the user experience.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the first input is voice input, and the first input includes violation description information. The method further includes: in response to obtaining the first input, obtaining the violation description information; and generating a violation report text based on the first data and the second data, including: generating a violation report text based on the violation description information, the first data, and the second data.

[0008] For example, the violation description information can be a user's description of the violating vehicle, the violation, or other information related to the violation report, such as vehicle color, model information, license plate number information, violation type, etc., including the user's voice in the violation description information, such as "The white car in front violated the rules" or "A car ran a red light".

[0009] Before generating a traffic violation report, a vehicle needs to process a large amount of data. Different vehicles have varying capabilities in acquiring and processing data, which may lead to the generation of incorrect violation information. Furthermore, when the user's violation description already includes some violation information, the computational load on the vehicle processing the data acquired by sensors can be reduced. Therefore, based on the above technical solution, when the user's description includes violation description information, this description information and the data acquired by sensors can be combined to generate the violation report text. This improves the accuracy of obtaining violation information and the efficiency of reporting violations, thereby enhancing the user experience.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: controlling the vehicle's display device to display multiple traffic violation information.

[0011] Based on the above technical solution, displaying traffic violation information makes it easier for users to confirm or modify the violation information, which helps improve the accuracy of obtaining violation information and thus enhances the user experience.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, multiple violation information includes first violation data, and the method further includes: in response to receiving user input to modify the first violation data, modifying the first violation data to second violation data.

[0013] For example, if the first violation data in multiple violation information is a license plate number, the license plate number could be "License Plate Number ABC123". In one scenario, when a user finds that the license plate number does not match the license plate number of the vehicle for which they want to report a violation, the first violation data "License Plate Number ABC123" can be changed to the second violation data "License Plate Number ABC124".

[0014] Based on the above technical solution, users can modify traffic violation information, which helps improve the accuracy of obtaining traffic violation information and thus enhances the user experience.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the first data and the second data are structured data. Based on the first data and the second data, a violation report text is generated, including: matching the location data in the first data with the coordinate data in the second data to obtain a matching result; determining the structured violation data based on the matching result, the first data, and the second data; and generating a violation report text based on the structured violation data.

[0016] For example, structured data can be information stored in a fixed format or schema. For instance, the structured data could be data in JavaScript object notation (JSON) format.

[0017] Based on the above technical solution, all types of data are structured, facilitating storage, analysis, and management. By matching and fusing data collected from various sensors, the accuracy of obtaining violation information and the efficiency of reporting violations can be improved, thereby enhancing the user experience.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, multiple violation information includes violation types. The method further includes: determining the violation type based on structured violation data; and generating a violation report text based on the violation type, the first data, and the second data.

[0019] Based on the above technical solution, different violation report texts can be matched when the type of violation is different, which helps to improve the accuracy of obtaining violation information and the efficiency of reporting violations, thereby improving the user experience.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: in response to obtaining the first input, obtaining the weather conditions of the area where the vehicle is located; generating a violation report text based on the violation type, the first data, and the second data, including: when the weather conditions are severe weather or nighttime, generating a violation report text based on the violation type, the first data, and at least one of the distance data, coordinate data, and speed data in the second data.

[0021] For example, severe weather includes heavy rain, heavy snow, dense fog, sandstorms, and other weather conditions with low visibility.

[0022] Based on the above technical solution, when the vehicle is in bad weather or at night, the images captured by the camera may be less clear. In such environments, at least one of the distance data, coordinate data, and speed data collected by the radar can be used as the benchmark for processing traffic violation information. This is beneficial to improving the accuracy of obtaining traffic violation information and thus enhancing the user experience.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the violation type is speeding. Based on the violation type, the first data, and the second data, a violation report text is generated, including: generating a violation report text based on the violation type, the category data and text data in the first data, and the speed data in the second data.

[0024] Radar-collected speed data is generally more accurate than that collected by cameras. Based on the above technical solution, in speeding scenarios, accurate speed information can be obtained by combining speed data collected by radar with category and text data collected by cameras. This improves the accuracy of obtaining violation information and thus enhances the user experience.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the violation type is solid line. Based on the violation type, the first data, and the second data, a violation report text is generated, including: generating a violation report text based on the violation type, the road environment data and category data in the first data, and the coordinate data in the second data.

[0026] Due to limitations in shooting angle, cameras may have blind spots when capturing images of vehicles violating traffic rules. This makes it difficult to accurately determine whether a vehicle has crossed a solid line, as well as the precise time and location of such crossing. Radar, on the other hand, typically has higher accuracy in locating vehicles or pedestrians than cameras. Therefore, based on the aforementioned technical solution, combining coordinate data collected by radar with road environment and category data collected by cameras in solid line crossing scenarios allows for a more accurate determination of the location and time of the crossing. This improves the accuracy of obtaining violation information and ultimately enhances the user experience.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, a traffic violation report text is generated based on the first data and the second data, including: determining multiple traffic violation information based on the first data, the second data, and a traffic violation report template for the area where the vehicle is located.

[0028] Different regions may have different requirements for the information provided when reporting traffic violations. Based on the above technical solution, multiple violation information can be specifically identified according to the violation reporting template for the vehicle's location. This avoids users having to manually filter violation information when reporting, which improves the efficiency of violation reporting and thus enhances the user experience.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, multiple violation information includes violation videos. The method further includes: obtaining violation videos based on the video stored in the dashcam, the time of obtaining the first input, the first data, and the second data; and obtaining at least one violation image based on the violation reporting template and the violation video.

[0030] When reporting traffic violations, corresponding evidence is required, which can be videos or pictures showing the violation. Based on the above technical solution, vehicles can obtain violation videos, and when the violation reporting template requires uploading images, it can extract violation images from the video, avoiding the need for users to manually crop images, thus improving the efficiency of violation reporting and enhancing the user experience.

[0031] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: in response to detecting a user report of a violation, sending multiple violation information.

[0032] Based on the above technical solution, multiple traffic violation information can be automatically sent to the traffic violation reporting platform, which helps to improve the efficiency of reporting traffic violations and thus improves the user experience.

[0033] Secondly, a traffic violation information acquisition device is provided, comprising: an acquisition unit for acquiring a first input from a user, the first input being used to instruct a vehicle to acquire traffic violation information; in response to acquiring the first input, acquiring first data collected by a camera device outside the vehicle's cabin and second data collected by at least one radar; a generation unit for generating a traffic violation report text based on the first and second data, the traffic violation report text including multiple pieces of traffic violation information, the multiple pieces of traffic violation information including at least one of the following: violation time, violation location, license plate number, license plate color, and violation type; and a control unit for controlling a prompting device of the vehicle to prompt the user with the traffic violation report text.

[0034] In conjunction with the second aspect, in some implementations of the second aspect, the first input is voice input, which includes violation description information. The acquisition unit is also used to acquire the violation description information in response to acquiring the first input. The generation unit is specifically used to generate a violation report text based on the violation description information, the first data, and the second data.

[0035] In conjunction with the second aspect, in some implementations of the second aspect, the control unit is also used to: control the vehicle's display device to display multiple violation information.

[0036] In conjunction with the second aspect, in some implementations of the second aspect, multiple violation information includes first violation data, and the device further includes: a modification unit, configured to modify the first violation data into second violation data in response to receiving user input to modify the first violation data.

[0037] In conjunction with the second aspect, in some implementations of the second aspect, the first data and the second data are structured data. The generation unit is specifically used to: match the location data in the first data with the coordinate data in the second data to obtain a matching result; determine the structured violation data based on the matching result, the first data, and the second data; and generate a violation report text based on the structured violation data.

[0038] In conjunction with the second aspect, in some implementations of the second aspect, multiple violation information, including violation types, are generated by the unit and are also used to: determine the violation type based on structured violation data; and generate a violation report text based on the violation type, the first data, and the second data.

[0039] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is also used to acquire the weather conditions of the area where the vehicle is located in response to acquiring the first input; the generation unit is specifically used to: when the weather conditions are severe weather or nighttime, generate a violation report text based on the violation type, the first data, and at least one of the distance data, coordinate data, and speed data in the second data.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the violation type is speeding. The generation unit is specifically used to generate a violation report text based on the violation type, the category data in the first data, the text data, and the speed data in the second data.

[0041] In conjunction with the second aspect, in some implementations of the second aspect, the violation type is solid line, and the generation unit is specifically used to: generate violation report text based on the violation type, road environment data and category data in the first data, and coordinate data in the second data.

[0042] In conjunction with the second aspect, in some implementations of the second aspect, the generation unit is specifically used to: determine multiple violation information based on the first data, the second data, and the violation reporting template of the area where the vehicle is located.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, multiple violation information includes violation videos, and the acquisition unit is also used to: acquire violation videos based on the video stored in the dashcam, the time of the first input, the first data, and the second data; and acquire at least one violation image based on the violation reporting template and the violation video.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the device further includes: a sending unit for sending multiple violation information in response to detecting a user report of a violation.

[0045] Thirdly, a device for obtaining traffic violation information is provided, the device comprising: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, so that the device performs a method corresponding to any of the implementations in the first aspect above.

[0046] Fourthly, a vehicle is provided, including a device corresponding to any of the implementations of the second or third aspect above.

[0047] Fifthly, a computer-readable storage medium is provided having instructions stored thereon that, when executed by a processor, cause the processor to implement a method corresponding to any of the implementations in the first aspect above.

[0048] Sixthly, a computer program product is provided, which includes computer program code that, when run on a computer, enables the computer to implement a method corresponding to any of the implementation methods in the first aspect above.

[0049] In a seventh aspect, a chip is provided, the chip including circuitry for performing a method corresponding to any of the implementations of the first aspect above. Attached Figure Description

[0050] Figure 1 This is a functional block diagram of the vehicle 100 provided in the embodiments of this application.

[0051] Figure 2 This is a schematic block diagram of the intelligent driving system provided in the embodiments of this application.

[0052] Figure 3 This is a schematic flowchart of the method 300 for obtaining traffic violation information provided in the embodiments of this application.

[0053] Figure 4 This is a schematic flowchart of a method 400 for obtaining traffic violation information according to an embodiment of this application.

[0054] Figure 5 This is a schematic diagram of a set of graphical user interfaces (GUIs) according to embodiments of this application.

[0055] Figure 6 This is a flowchart illustrating the process of obtaining traffic violation information.

[0056] Figure 7 This is a flowchart illustrating a method 700 for obtaining structured video information according to an embodiment of this application.

[0057] Figure 8This is a flowchart illustrating a method for capturing traffic violation videos based on the user's initial input and stored data.

[0058] Figure 9 This is a schematic diagram of a set of graphical user interfaces according to an embodiment of this application.

[0059] Figure 10 This is a schematic diagram of a set of graphical user interfaces according to an embodiment of this application.

[0060] Figure 11 This is a schematic diagram of a set of graphical user interfaces according to an embodiment of this application.

[0061] Figure 12 This is a schematic diagram of a set of graphical user interfaces according to an embodiment of this application.

[0062] Figure 13 This is a schematic diagram of a set of graphical user interfaces according to an embodiment of this application.

[0063] Figure 14 This is a schematic block diagram of the violation information acquisition device 1400 provided in the embodiments of this application. Detailed Implementation

[0064] The technical solutions of this application will now be described with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. "At least one" refers to one or more. For example, "at least one of A and B," similar to "A and / or B," describes the relationship between related objects, indicating that three relationships can exist. For example, at least one of A and B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0065] The prefixes such as "first" and "second" used in this application embodiment are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this application embodiment does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not constitute unnecessary restrictions due to the use of such prefixes. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0066] Figure 1This is a functional schematic diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 may include a sensing system 110, a computing platform 120, and a display device 130. The sensing system 110 may include one or more sensors for sensing information about the environment surrounding the vehicle 100. For example, the sensing system 110 may include a positioning system, which may be a Global Positioning System (GPS), a BeiDou Navigation Satellite System, or another positioning system. As another example, the sensing system 110 may include one or more of the following: an inertial measurement unit (IMU), an accelerometer, a lidar, a millimeter-wave radar, an ultrasonic radar, and a camera device. Additionally, the sensing system 110 may also include a voice interaction system.

[0067] Some or all of the functions of vehicle 100 can be controlled by computing platform 120. Computing platform 120 may include one or more processors, such as processors 121 to 12n (n being a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement some or all of the functions of the aforementioned units. Furthermore, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. In addition, the computing platform 120 may also include a memory for storing instructions. Some or all of the processors 121 to 12n can call the instructions in the memory to implement the corresponding functions.

[0068] The in-cabin display devices 130 are mainly divided into two categories: the first is the in-vehicle display screen; the second is the projection display screen, such as the head-up display (HUD). An in-vehicle display screen is a physical display screen and an important component of the in-vehicle infotainment system. Multiple displays can be installed in the cabin, such as the digital instrument cluster display, the central control screen, the display screen in front of the front passenger (also known as the front-seat passenger), the display screen in front of the left rear passenger, the display screen in front of the right rear passenger, and even the car window can be used as a display screen. A head-up display, also known as a head-up display system, is mainly used to display driving information such as speed and navigation on a display device in front of the driver (such as the windshield). This reduces the driver's eye-shift time, avoids pupil changes caused by eye-shifting, and improves driving safety and comfort. Examples of HUDs include combiner-HUD (C-HUD) systems, windshield-HUD (W-HUD) systems, and augmented reality HUD (AR-HUD) systems. It should be understood that HUDs can also evolve into other types of systems as technology progresses, and this application does not limit them.

[0069] The above description of the display device 130 uses an in-vehicle display screen and a projection display screen as examples, but the embodiments of this application are not limited thereto. For example, the display device 130 can also be a light display screen or a projection screen.

[0070] Optionally, the structure of the vehicle 100 described above is merely illustrative. In actual applications, various components of the vehicle 100 may be added or removed as needed.

[0071] Vehicle 100 may include an intelligent driving system, which may include an advanced driving assistant system (ADAS) and an autonomous driving system (ADS). The intelligent driving system uses various sensors on the vehicle (including but not limited to: lidar, millimeter-wave radar, camera devices, ultrasonic sensors, global positioning system, inertial measurement unit) to acquire information from the vehicle's surroundings, and analyzes and processes the acquired information to achieve functions such as obstacle perception, target recognition, vehicle positioning, path planning, and driver monitoring / alerts, thereby improving the safety, automation, and comfort of driving the vehicle.

[0072] As a core technology of intelligent driving, voice interaction systems free up the driver's hands, allowing them to better focus on driving. Given the frequent occurrence of traffic violations, combining the vehicle's voice interaction system with violation information collection could expand the functionality of the voice interaction system and facilitate drivers in recording detailed descriptions of violations, thus improving the user experience.

[0073] Therefore, this application provides a method and apparatus for obtaining traffic violation information, which combines the acquisition of traffic violation information with the voice interaction system of a smart cockpit, expanding the functionality of voice interaction. The method and apparatus for obtaining traffic violation information of this application can be used to assist police vehicles in obtaining traffic violation information, and can also be used by private car owners to facilitate users in reporting traffic violations, thereby improving road traffic safety.

[0074] For example, Figure 2 A schematic block diagram of an intelligent driving system provided in an embodiment of this application is shown. The vehicle 100 in this system includes a perception system 110, an information fusion system 220, a control system 230, and vehicle hardware. The vehicle hardware may include a display device, speakers, microphones, etc. In this embodiment, the perception system 110 can acquire information about the surrounding environment and other vehicles, and send it to the information fusion system 220 for further processing; alternatively, it can upload the relevant information to the cloud for computation. Based on the information acquired by the perception system 110, the information fusion system 220 further extracts specific traffic violation information and sends it to the control system 230. Based on the traffic violation information received from the information fusion system 220, the control system 230 controls the vehicle; for example, the control system 230 can control the vehicle's display device to display the corresponding traffic violation information.

[0075] Optionally, the vehicle can also transmit relevant information via the network to electronic devices connected to the vehicle's infotainment system (e.g., mobile phones, tablets, personal computers) for the user to synchronize task progress. Alternatively, the user can also use this electronic device to control the vehicle to perform corresponding traffic violation information collection operations.

[0076] The information fusion system 220 and the control system 230 mentioned above can be located in the computing platform 120.

[0077] Figure 3A schematic flowchart of a traffic violation information acquisition method 300 provided in an embodiment of this application is shown. This method 300 can be executed by the vehicle 100, or by the computing platform 120, or by a system-on-a-chip (SoC) within the computing platform 120, or by a processor, chip, or circuit within the computing platform 120, or by the intelligent driving system. The following embodiments use an intelligent driving system as an example for illustration. The method 300 includes:

[0078] S310: Obtain first input from the user, which is used to instruct the vehicle to obtain traffic violation information.

[0079] For example, the first input could be the user's voice, text entered by the user through a display device, or the user's action of clicking a button or control related to obtaining violation information or reporting violations.

[0080] For example, a corresponding button or switch can be set up to determine whether the user needs to summarize traffic violation information based on the user's operation. This button can be a physical button or a virtual control displayed on the screen. For instance, a control related to traffic violation reporting can be set up in the corresponding application (APP) or mini-program. When the user clicks this control, it is determined that the user has instructed the vehicle to obtain traffic violation information. This control can be displayed on the vehicle's display device, or it can be displayed on the display device of the electronic device when the user connects the electronic device to the vehicle's system. When the user connects the electronic device to the vehicle's system, obtaining the user's first input also includes obtaining the user's first input to the electronic device. For example, a control for obtaining traffic violation information can be set up on the mobile phone connected to the vehicle's system. When the user clicks this control on the mobile phone, the vehicle obtains the user's first input through the mobile phone.

[0081] Taking voice input as an example, users can directly express commands related to obtaining or reporting traffic violations. For instance, the first input could be a user's voice saying, "A car has violated traffic rules, collect violation information," or "Report this white car that ran a red light." Alternatively, the first input could be other descriptions related to obtaining or reporting traffic violations, such as "A car has violated traffic rules, make a note of it," or "This car is driving too fast, is it speeding?" In one implementation, when the user's first input is voice input, the user's voice information can be recognized first, converted into text, and then the intent recognition of the text can be performed using a large language model (LLM). Speech-to-text conversion can be implemented using automatic speech recognition (ASR) algorithms, or it can access an existing database of large language models by calling an application programming interface (API).

[0082] When the first input is the user's voice or text entered by the user through a display device, the intent of the voice or text can be recognized to determine whether the first input instructs the vehicle to obtain traffic violation information. If the first input is the user clicking a button or control related to obtaining traffic violation information or reporting a traffic violation, upon detecting the user clicking such a button or control, it can be directly determined that the first input instructs the vehicle to obtain traffic violation information.

[0083] In this embodiment, traffic violation information refers to all information related to reporting traffic violations, including violation videos or images, violation types (one or more), license plate numbers, license plate colors, vehicle model information, violation time, violation duration, violation location, weather conditions, relevant regulations and traffic laws and regulations of the region where the violation occurred, personal information related to the reporter, the reporter's description, or text descriptions generated by the vehicle's system. The relevant regulations, traffic laws and regulations of the region where the violation occurred, and the personal information related to the reporter can be stored on the vehicle or in the cloud before obtaining the user's initial input. Weather conditions include sunny, cloudy, rainy, snowy, foggy, and sandstorm weather, as well as light information, such as daytime, nighttime, and tunnel conditions. Weather conditions can be obtained through weather sensors or by recognizing environmental information in images or videos captured by camera devices; this embodiment does not limit the method of obtaining weather conditions.

[0084] S320: In response to receiving a first input, acquire first data collected by a camera outside the vehicle's cabin and second data collected by at least one radar.

[0085] Cameras and radar can be Figure 1 and Figure 2 The radar in the sensing system 110 shown may include millimeter-wave radar, lidar, ultrasonic radar, etc.

[0086] For example, acquiring first data collected by a camera outside the vehicle's cabin and second data collected by at least one radar can represent first and second data stored by the vehicle before acquiring the first input, or first and second data collected by the vehicle during the time period of acquiring the first input, or first and second data collected by the vehicle during a period of time after acquiring the first input.

[0087] S330: Based on the first data and the second data, generate a traffic violation report text. The traffic violation report text includes multiple pieces of traffic violation information, including at least one of the following: violation time, violation location, license plate number, license plate color, and violation type.

[0088] Optionally, before generating the violation report text, the process may further include: obtaining multiple violation information based on the first data and the second data. The violation information can be obtained in real time or based on stored data, as detailed in S430 below.

[0089] Traffic violation report text can be a conversational description including violation information, which can be summarized using an LLM (Local Management Model). The report text can simply state the complete violation information, or it can include a question asking the user whether they want to report the violation. For example, based on the user's voice command "The white car on the left has violated traffic rules, please note it," after determining the violation type to be crossing a solid line, and combining multiple violation details with the preset prompts in the LLM, a complete text description can be generated. The report text could be: "The white vehicle with license plate ABC124 on the left illegally crossed a solid line while changing lanes from the third lane to the second lane from the left, and was 20 meters away from this vehicle," or "The white vehicle with license plate ABC124 on the left illegally crossed a solid line while changing lanes from the third lane to the second lane from the left, and was 20 meters away from this vehicle. Would you like to report it now?"

[0090] Optionally, the first input is voice input, which includes a description of the violation. The method further includes: in response to obtaining the first input, obtaining the description of the violation; and generating a violation report text based on the first data and the second data, including: generating the violation report text based on the description of the violation, the first data, and the second data.

[0091] For example, the violation description information and multiple violation information from the first input can be merged according to priority, and then summarized and generated into a violation report text using LLM. The violation description information can be a user's description of the violating vehicle, the violating behavior, or other information related to the violation report, such as vehicle color, model information, license plate number information, violation type, etc., including the user's voice description of the violation, such as "The white car in front violated the rules" or "A car ran a red light".

[0092] In one possible scenario, when the violation description information is inconsistent with one of multiple violation information, the priority of the violation description information can be higher than the violation information obtained by processing data from the vehicle's sensors. For example, if the user's first input is "The white car with license plate ABC124 on the left committed a violation, please note it," and the license plate number obtained by processing the first and second data is ABC123, then the final generated violation report text could be "The white car with license plate ABC124 on the left illegally crossed the solid line while changing lanes from the third lane to the second lane from the left, and was 20 meters away from this vehicle."

[0093] Taking speeding detection as an example, speed limit signs typically appear only at intervals, making it difficult to detect speeding in areas without such signs. In one scenario, a user's first input might be, "The speed limit on this road is 120 km / h, the black car next to me is definitely violating the rules." This first input includes violation descriptions such as "speed limit 120" and "black." Based on this violation description, the first data, and the second data, the generated violation description text could be, "The black vehicle with license plate ABC124 on the left is speeding at 130 km / h."

[0094] Optionally, the first data and the second data can be structured data. Generating a violation report text based on the first data and the second data includes: matching the location data in the first data with the coordinate data in the second data to obtain a matching result; determining structured violation data based on the matching result, the first data, and the second data; and generating a violation report text based on the structured violation data.

[0095] For example, structured data can be data stored in a fixed format or pattern, such as JSON format data. The first data can be video structured data, and the second data can be LiDAR structured data and / or millimeter-wave radar structured data. The positional data in the first data can be bounding box position data or pixel group coordinate data, and the coordinate data in the second data can be two-dimensional coordinate data.

[0096] When fusing the first and second data, the position data in the first data and the coordinate data in the second data can be transformed according to the installation location of the camera device. The matching relationship between objects is determined based on the transformed coordinates. Then, the first and second data are fused according to the matching relationship to obtain complete structured violation data. A violation report text is generated based on this structured violation data. Taking vehicle A as an example, the first data includes the vehicle's pixel coordinate data B, license plate number C, and vehicle color D. The second data includes the vehicle's two-dimensional coordinate data E and speed data F. Based on the pixel coordinate data B and the two-dimensional coordinate data E, it can be determined that the first and second data match the same vehicle A. The structured violation data includes pixel coordinate data B, license plate number C, vehicle color D, two-dimensional coordinate data E, and speed data F.

[0097] Optionally, the multiple violation information may include violation types, and the method may further include: determining the violation type based on structured violation data; and generating a violation report text based on the violation type, the first data, and the second data.

[0098] For example, based on the coordinate data of objects categorized as "pedestrians" and "vehicles" in structured traffic violation data, it can be determined whether the distance between the vehicle and the pedestrian is less than or equal to a preset distance; when this distance is less than or equal to the preset distance, it can be determined that the vehicle failed to yield to the pedestrian. Based on the position data of objects categorized as "vehicles" and "solid lane lines" in structured traffic violation data, when these two position data overlap, it can be determined that the vehicle crossed the solid line. As another example, based on textual data obtained from the first data set (e.g., speed limit requirements) and speed data from the second data set, it can be determined whether the vehicle was speeding.

[0099] Different violation report texts can be generated based on different conditions when the vehicle is driven in different environments or when the type of violation is different.

[0100] Optionally, the method may further include: in response to obtaining the first input, obtaining the weather conditions of the area where the vehicle is located; generating a violation report text based on the violation type, the first data, and the second data, including: when the weather conditions are severe weather or nighttime, generating a violation report text based on the violation type, the first data, and at least one of the distance data, coordinate data, and speed data in the second data.

[0101] For example, severe weather includes heavy rain, heavy snow, dense fog, sandstorms, and other weather conditions with low visibility.

[0102] Both the first and second data can include at least one of distance, coordinate, and speed data. For example, in inclement weather such as rain, snow, or fog, or in low-light conditions at night, it is difficult for the camera to obtain accurate distance data. When corresponding data in the first and second data differ, the second data can be prioritized as the basis for generating the violation report. For example, by combining the position and orientation of pedestrians or vehicles, as well as distance data, the exact time and location of a collision can be determined.

[0103] Optionally, when the violation type is speeding, a violation report text is generated based on the violation type, the first data, and the second data. This may include generating the violation report text based on the violation type, the category data in the first data, the text data, and the speed data in the second data.

[0104] Radar-collected speed data is generally more accurate than that collected by cameras. When obtaining information related to speeding violations, radar-collected speed data can be prioritized over speed data obtained by cameras, thus facilitating the generation of more accurate violation reports.

[0105] Optionally, when the violation type is solid line violation, a violation report text is generated based on the violation type, the first data, and the second data. This may include generating the violation report text based on the violation type, the road environment data and category data in the first data, and the coordinate data in the second data.

[0106] For example, the images captured by the camera device are limited by the shooting angle, and there is partial overlap or blind spots between the vehicle and the lane lines in the image, which is not conducive to determining the time of the violation. In this case, the location and specific time of the vehicle crossing the solid line can be determined by using the road environment data in the first data (including the solidity or darkness of the lane lines, the position or width of the lane lines, etc.) and the coordinate data in the second data, which is helpful for generating a violation report text.

[0107] Optionally, generating a traffic violation report text based on the first data and the second data may include: determining multiple violation information based on the first data, the second data, and a traffic violation report template for the area where the vehicle is located.

[0108] Different regions may have different requirements for the necessary information to report traffic violations. A traffic violation reporting template can be built based on the reporting requirements of each region, and this template can be stored on the vehicle or in the cloud. For example, before determining multiple violations, the vehicle's location information can be obtained to determine the area where the vehicle is located. Then, based on the first set of data, the second set of data, and the traffic violation reporting template for the vehicle's location, multiple violations can be determined. This reduces the user's need to filter violation information when reporting, enabling more targeted reporting.

[0109] Optionally, the area where the vehicle is located can also be provided by the user. For example, the user's first input may include information related to the area where the vehicle is located. In response to receiving the user's first input, information about the area where the vehicle is located can be obtained, thereby determining multiple corresponding traffic violation information based on the traffic violation reporting template for that area.

[0110] Optionally, multiple traffic violation information may include traffic violation videos. The method further includes: determining the traffic violation video based on the video stored in the dashcam, the time of obtaining the first input, the first data, and the second data; and determining at least one traffic violation image based on the traffic violation reporting template and the traffic violation video.

[0111] For methods of extracting violation videos, please refer to S431 or S432 below. Since some regions require uploading images when reporting violations and cannot upload videos, at least one violation image can be extracted from the violation video automatically or manually. Automatic extraction can be done at equal intervals based on the required number of violation videos and images.

[0112] S340: The vehicle control device displays a traffic violation report text to the user.

[0113] For example, the notification device could be a speaker in the vehicle or an electronic device connected to the vehicle. For instance, the vehicle could be controlled to read out a traffic violation report via voice, and this voice reading function could be implemented using text-to-speech (TTS) technology.

[0114] For example, the notification device can be a display device of a vehicle or an electronic device connected to the vehicle. This display device can be a screen or projection device inside the vehicle's cabin, or a screen of a user's electronic device connected to the vehicle. For instance, the vehicle can be controlled to display a traffic violation report text on a screen.

[0115] Optionally, the method may further include: controlling the vehicle's display device to display multiple traffic violation information.

[0116] Optionally, the multiple violation information includes first violation data, and the method may further include: in response to receiving user input to modify the first violation data, modifying the first violation data to second violation data.

[0117] For example, after the prompting device displays the violation report text to the user and / or the display device displays multiple violation information, if the user needs to modify the violation information, they can change the first violation data among the multiple violation information to the second violation data. For instance, if the license plate number obtained by processing the first and second data is ABC123, but the actual license plate number of the vehicle to be reported is ABC124, and the user finds that the license plate number is incorrect, they can change the license plate number to ABC124 through voice, text input, or other means.

[0118] Optionally, the method may further include: sending multiple violation information in response to detecting a user report of a violation.

[0119] For example, user actions for reporting traffic violations include voice commands, text input, and clicking controls or buttons related to the violation report. In response to detecting a user's violation report, multiple violation information entries can be automatically sent to the violation reporting platform to complete the report. Optionally, when a user connects their electronic device to the vehicle's infotainment system, they can also send multiple violation information entries through the electronic device to report the violation.

[0120] According to the above scheme, the method 400 for obtaining traffic violation information can improve the accuracy of obtaining traffic violation information and increase the efficiency of reporting traffic violations, thereby improving the user experience.

[0121] Figure 4 This diagram illustrates a schematic flowchart of a method 400 for obtaining traffic violation information according to an embodiment of this application. Figure 4 As shown, the method 400 of this application embodiment includes:

[0122] S410: Get the first input from the user.

[0123] Optionally, based on the user's first input, structured text description information can be obtained. For example, when the first input is "Remember that red car that ran a red light, the license plate is ABC123", the corresponding structured text description information may include: {"violation license plate":"ABC123","violation car color":"red","violation behavior":"running a red light"}; when the first input is "A car violated the rules, remember it", the corresponding structured text description information may be {}, indicating that the first input only includes the intention to record the violation information, without providing any other information.

[0124] Optionally, while acquiring the first input, the time and location information of the first input can be recorded, and the content, time, and location information of the first input can be combined to form a text description structured information. The location information can be obtained by combining the positioning system and calling the navigation interface, and the time information can be obtained by calling the time interface in the operating system. For example, when the first input is "Remember the red car that ran a red light to the left, the license plate is ABC123", the text description structured information composed of the content, time, and location information of the first input can include: {"Violation time":"2023-10-15T14:30:00","Violation license plate":"ABC123","Violation vehicle color":"Red","Violation behavior":"Running a red light","Relative position":"Left front","Location information":"Intersection in Xuhui District, Shanghai","Severity":"High"}.

[0125] Optionally, the severity of the violation can be determined based on the type of violation described by the user. For example, for violations that seriously affect traffic safety, such as running red lights and speeding, the structured text description can include {"severity":"high"}; for violations such as crossing solid lines, the structured text description may not include severity information, or may mark the severity as low.

[0126] S420: Determine whether the user needs to obtain traffic violation information.

[0127] For example, when a user needs to obtain traffic violation information, step S430 is executed; otherwise, step S410 is executed.

[0128] Upon receiving the user's initial input, and determining that the user desires traffic violation information, the traffic violation information collection process can be initiated. For ease of description, the vehicle for which traffic violation judgment needs to be made will be referred to as the target vehicle.

[0129] For example, a user in the cabin can issue a voice command, "The vehicle in front has committed a traffic violation, please record it." The vehicle's voice interaction system will recognize the user's voice, identify the "vehicle in front" as the target vehicle, and initiate the violation information collection process. Optionally, after confirming that the user needs to collect violation information, the system can also reply via voice, "Okay, collecting violation information for you."

[0130] Figure 5 This diagram illustrates a set of graphical user interfaces according to embodiments of this application. The content displayed on the screen in the vehicle display device 130 can be as follows: Figure 5As shown in (a), the display interface includes a content display area 510 and a function bar 520. Exemplarily, the content display area 510 may include: icons for multiple applications, each icon used to activate an application, such as settings, navigation, traffic violation reporting, smart driving, weather, maps, app store, etc.; and Bluetooth, Wi-Fi, and cellular signal icons, used to indicate the vehicle's Bluetooth on and / or connection status, Wi-Fi on status, and cellular signal strength, respectively. The function bar 520 includes a home icon and volume controls, used to control the display of the home page in the content display area 510 and to adjust the volume of the sound device, respectively.

[0131] Optionally, in response to the user clicking the traffic violation reporting control 501, the vehicle can display the following on the screen: Figure 5 The display interface shown in (b) includes a traffic violation information collection control 502. Optionally, the display interface can also display information about the vehicle's location, allowing the user to perform corresponding operations based on the requirements for reporting traffic violations in different regions. When the user clicks on the traffic violation information collection control 502, it is determined that the user wants to collect traffic violation information, and the traffic violation information collection process can begin.

[0132] S430: Obtain traffic violation information.

[0133] Figure 6 A schematic diagram illustrating a process for obtaining traffic violation information is shown. In one embodiment, after acquiring sensor data (including first data and second data) from multiple sensors in the perception system 110, the vehicle's intelligent driving computing platform 601 can process this sensor data to obtain reference object data. Reference objects include surrounding vehicles, pedestrians, traffic lights, traffic signs, roads, etc. The reference object data can be structured data. The intelligent driving computing platform 601 can be a mobile data center (MDC) platform, located either on the vehicle or in the cloud.

[0134] For example, visual image processing can be performed on images or videos captured by camera devices to obtain structured video data, including traffic light color data, traffic light timing data, vehicle travel direction within the road, vehicle and pedestrian position and movement direction, lane lines and ground markings, content of various signs (such as speed limit signs, turn signs, etc.), vehicle color, license plate number, and license plate color; structured LiDAR data can be obtained from point cloud data collected by LiDAR, such as distance data between vehicles and reference objects, road structure data, and reference object position data; structured millimeter-wave radar data can be obtained from transmitted and received signal data, such as distance data between vehicles and reference objects, speed data of other vehicles or pedestrians, and angle (direction) data of other vehicles; optionally, distance data between vehicles and reference objects can be obtained from transmitted and received sound wave data by ultrasonic radar.

[0135] Based on reference data obtained by the intelligent driving computing platform 601 and the user's initial input, the intelligent cockpit platform 602 can combine the reference data, the user's initial input, and traffic laws and regulations to make a violation judgment and output violation information. The intelligent cockpit platform 602 can be a cockpit domain controller (CDC). For example, by combining the road traffic laws and regulations of the area where the vehicle is located, the driving data of the target vehicle identified by the perception system, and the surrounding environmental data, the type of violation of the target vehicle can be determined. The violation information generated by the intelligent cockpit platform 602 may include the violation type, violation time, and violation video, etc.

[0136] Based on the user's initial input and sensor data acquired by multiple sensors in the perception system 110, multiple traffic violation information can be obtained. In this embodiment, the moment the user's initial input is acquired is referred to as the first time. For example, when a user issues a voice command, "The vehicle ahead has committed a traffic violation, record it," the time when the vehicle receives the user's voice command (i.e., the moment the user finishes speaking) is determined as the first time. When a user inputs text using a display device, the first time can be the moment the user finishes typing, or the moment the user clicks an "OK," "Search," or similar control after typing. When a user clicks a button or control, the first time is the moment the user clicks that button or control. The vehicle's dashcam can record video for a period of time prior to the first time.

[0137] There are two possible methods for obtaining traffic violation information based on sensor data and video from a dashcam: real-time acquisition (method 431) and acquisition based on stored data (method 432). Real-time acquisition refers to performing real-time calculations and processing of sensor data and video from a dashcam while simultaneously receiving the user's initial input. Method 431 for real-time acquisition of traffic violation information may include the following steps:

[0138] S431-1: Acquire data from multiple reference objects based on sensor data.

[0139] For example, reference data includes structured video data, structured LiDAR data, and structured millimeter-wave radar data. Figure 7 This diagram illustrates a flowchart of a method 700 for acquiring structured video data according to an embodiment of this application. Figure 7 As shown, the method 700 may include S710: acquiring first video structured data in a first video image, the first video structured data including location data and category data.

[0140] For example, the first video image may be a frame captured by a camera outside the vehicle cabin at the first time corresponding to the detection of the user's first input. In one implementation, object detection can be performed on objects in the first video image, including detecting the object's position and category. The position can be represented by an outer bounding box (also called a "box") covering the object's outline, and the category can be obtained by inputting the first video image into a preset model; the specific category can be set in the model's training data. Boxes 1 through 7 below illustrate an example of structured data for the first video image:

[0141] [{"Box1":[100,150,200,250],"Category":"Traffic Lights"},

[0142] {"Box2":[300,350,400,450],"Category":"Vehicles"},

[0143] {"Box3":[500,550,600,650],"Category":"Pedestrian"},

[0144] {"Box4":[700,750,800,850],"Category":"Signage"},

[0145] {"Box 5":[900,950,1000,1050],"Category":"Billboard"},

[0146] {"Box 6":[1100,1150,1200,1250],"Category":"Delta Line"},

[0147] {"Box 7":[1300,1350,1400,1450],"Category":"Fire Hydrant"}...]

[0148] Taking the object corresponding to "Box 1" as an example, [100,150,200,250] can represent the box position data of the object, and "traffic light" can represent the category data of the object.

[0149] S720: Based on the first video structured data, obtain the second video structured data, which includes the first video structured data and at least one of pixel group coordinate data, color data, and text data.

[0150] For example, the second video structured data may include the first video structured data, and at least one of pixel group coordinate data, color data, and text data may be added to the first video structured data.

[0151] For example, the first video image can be object segmented to obtain the accurate pixel group position of the object corresponding to each box according to the image segmentation method. Additionally, the first video image is input into a color recognition module and / or a text recognition module to further recognize colors and text for specific categories in the first video image, such as traffic lights, vehicles, etc. For example, color data is recognized for objects classified as traffic lights, color data and text data (e.g., license plate numbers) are recognized for objects classified as vehicles, and text data (e.g., the text content displayed on the sign) is recognized for objects classified as signs. Boxes 1 through 7 below illustrate an example of second video structured data:

[0152] [{"Box1":[100,150,200,250],"Category":"Traffic Light","Color":"Red","Pixel Group Coordinates":[(100,150),(100,250),(200,250),(200,150)]},

[0153] {"Box2":[300,350,400,450],"Category":"Vehicle","Color":"White","Pixel Group Coordinates":[(300,350),(300,450),(400,450),(400,350)]},

[0154] {"Box3":[500,550,600,650],"Category":"Pedestrian","Pixel Group Coordinates":[(500,550),(500,650),(600,650),(600,550)]},

[0155] {"Box4":[700,750,800,850],"Category":"Sign","Text":"No Parking","Pixel Coordinates":[(700,750),(700,850),(800,850),(800,750)]},

[0156] {"Box 5":[900,950,1000,1050],"Category":"Billboard","Pixel Group Coordinates":[(900,950),(900,1050),(1000,1050),(1000,950)]},

[0157] {"Box 6":[1100,1150,1200,1250],"Category":"Dark Lane Line","Pixel Group Coordinates":[(1100,1150),(1100,1250),(1200,1250),(1200,1150)]},

[0158] {"Box 7":[1300,1350,1400,1450],"Category":"Fire Hydrant","Pixel Group Coordinates":[(1300,1350),(1300,1450),(1400,1450),(1400,1350)]}...]

[0159] For certain specific categories of objects, such as "traffic lights" corresponding to "box 1", "vehicles" corresponding to "box 2", and "signs" corresponding to "box 4", the second video structured data adds color and text data. Taking the object corresponding to "box 1" as an example, compared with the first video structured data in S710, the second video structured data adds pixel group coordinate data represented by [(100,150),(100,250),(200,250),(200,150)] and color data represented by "red". Taking the object corresponding to "box 4" as an example, compared with the first video structured data in S710, the second video structured data adds pixel group coordinate data represented by [(700,750),(700,850),(800,850),(800,750)] and text data represented by "No parking".

[0160] Optionally, the "billboard" corresponding to "box 5" may also have color and text, but the color and text in the billboard are not related to the judgment of vehicle violations. In order to reduce the amount of calculation, the color data and text data of objects in the category of "billboard" can be omitted.

[0161] S730: Obtain the third video structured data from the second video image.

[0162] For example, the second video image may be another frame captured by a camera outside the vehicle cabin after the first video image has been acquired, at a preset number of frames or a preset time interval from the first video image. The data categories contained in the third video structured data may be consistent with those in the second video structured data. The method for acquiring the third video structured data is described in S710 and S720, and will not be repeated here.

[0163] S740: Based on the second and third video structured data, obtain fourth video structured data, which includes the second video structured data and speed data, or includes the third video structured data and speed data.

[0164] For example, the category data, color data, and text data in the second and third video structured data are consistent, while the bounding box position data and pixel group coordinate data may be the same or different. For instance, the bounding box position data and pixel group coordinate data of moving objects such as vehicles and pedestrians may differ in the first and second video images, while the bounding box position data and pixel group coordinate data of fixed objects such as fire hydrants, solid lane lines, and signs may be the same in the first and second video images.

[0165] For example, for moving objects such as vehicles and pedestrians, based on the box position data or pixel group coordinate data in the second and third video structured data, combined with the time interval between the first and second video images and the vehicle's speed, the corresponding vehicle or pedestrian speed data in the video can be obtained. The following example of a fourth video structured data is shown based on the second video structured data and speed data from "Boxes 1" to "Boxes 7":

[0166] [{"Box1":[100,150,200,250],"Category":"Traffic Light","Color":"Red","Pixel Group Coordinates":[(100,150),(100,250),(200,250),(200,150)]},

[0167] {"Box2":[300,350,400,450],"Category":"Vehicle","Color":"White","Pixel Group Coordinates":[(300,350),(300,450),(400,450),(400,350)],"Video Speed":100},

[0168] {"Box 3":[500,550,600,650],"Category":"Pedestrian","Pixel Group Coordinates":[(500,550),(500,650),(600,650),(600,550)],"Video Speed":2.5},

[0169] {"Box4":[700,750,800,850],"Category":"Sign","Text":"No Parking","Pixel Coordinates":[(700,750),(700,850),(800,850),(800,750)]},

[0170] {"Box 5":[900,950,1000,1050],"Category":"Billboard","Pixel Group Coordinates":[(900,950),(900,1050),(1000,1050),(1000,950)]},

[0171] {"Box 6":[1100,1150,1200,1250],"Category":"Dark Lane Line","Pixel Group Coordinates":[(1100,1150),(1100,1250),(1200,1250),(1200,1150)]},

[0172] {"Box 7":[1300,1350,1400,1450],"Category":"Fire Hydrant","Pixel Group Coordinates":[(1300,1350),(1300,1450),(1400,1450),(1400,1350)]}...]

[0173] Compared to the second video structured data in S720, the fourth video structured data in boxes "1" to "7" adds speed data for objects that may be in motion, such as pedestrians and vehicles. Taking the object corresponding to "box 2" as an example, the second video structured data, compared to the second video structured data in S720, adds speed data represented by "speed in video":100.

[0174] It should be understood that the numbers “Box 1” to “Box 7” are for descriptive convenience only. In actual applications, the objects identified in the image do not need to be sorted or numbered.

[0175] The intelligent driving platform 601 can obtain data such as the position, category, and distance of an object based on the object contour composed of point cloud data acquired by the LiDAR. In one embodiment, the height and distance of the object can be projected onto a first plane to obtain LiDAR structured data. The first plane can be the plane where the height of the LiDAR sensor outside the vehicle's cabin is located at the first moment, or the plane where the vehicle is located on the ground, or other planes parallel to the plane where the vehicle is located on the ground at the first moment, etc. This application does not limit the height of the first plane. The coordinates of the vehicle in the first plane are [0,0], and the height is 0. The following shows an example of LiDAR structured data:

[0176] [{"Coordinates":[100,150],"Laser_ObjectID":1,"Height":600,"Distance":1500},

[0177] {"Coordinates":[200,250],"Laser_ObjectID":2,"Height":120,"Orientation Vector":[1,0],"Distance":800},

[0178] {"Coordinates":[300,350],"Laser_ObjectID":3,"Height":170,"Orientation Vector":[0,1],"Distance":500},

[0179] {"Coordinates":[400,450],"Laser_ObjectID":4,"Height":230,"Distance":1200},

[0180] {"Coordinates":[500,550],"Laser_ObjectID":5,"Height":300,"Distance":950},

[0181] {"Coordinates":[700,750],"Laser_ObjectID":6,"Height":30,"Distance":400}...]

[0182] For example, LiDAR structured data may include various types of data such as coordinate data, identification data (e.g., object identifier (ID), which can be represented by numbers), height data, and distance data. For objects in a moving state, LiDAR structured data may also include orientation data (e.g., orientation vector, representing the orientation between the object and the vehicle).

[0183] The intelligent driving platform 601 can acquire data such as the position, velocity, and distance of an object based on signal data obtained from millimeter-wave radar. In one embodiment, the millimeter-wave radar can use the same first plane as lidar to represent the object's position, velocity, and distance data in coordinates on the first plane. The following is an example of structured data from millimeter-wave radar:

[0184] [{"Coordinates":[100,150],"Millimeter Wave_Object ID":1,"Distance":1500},

[0185] {"Coordinates":[200,250],"Millimeter Wave_Object ID":2,"Orientation Vector":[1,0],"Velocity":25,"Acceleration":2,"Distance":800},

[0186] {"Coordinates":[300,350],"Millimeter Wave_Object ID":3,"Orientation Vector":[0,1],"Velocity":1.5,"Acceleration":0,"Distance":500},

[0187] {"Coordinates":[400,450],"Millimeter Wave_Object ID":4,"Distance":1200},

[0188] {"Coordinates":[500,550],"Millimeter Wave_Object ID":5,"Distance":950},

[0189] {"Coordinates":[700,750],"Millimeter Wave_Object ID":6,"Distance":400},

[0190] {"Coordinates":[1700,1750],"Millimeter Wave_Object ID":7,"Distance":16000}...]

[0191] For example, millimeter-wave radar structured data may include various types of data such as coordinate data, identification data, distance data, velocity data, and acceleration data. For moving objects, millimeter-wave radar structured data may also include orientation data.

[0192] S431-2: Integrating data from multiple reference points.

[0193] Millimeter-wave radar possesses strong resistance to low-frequency interference and the ability to penetrate fog, smoke, and dust, enabling precise determination of the relative distance, relative speed, and orientation between a target object and a vehicle. However, it struggles to obtain accurate object shape data. Point cloud data can form the outline of an object, but it is difficult to determine the object's category. Therefore, after acquiring reference object data, data from different sensors or multiple structured data sets from the same sensor can be fused to obtain accurate violation information. For example, vehicle speed can be acquired using millimeter-wave radar; combining this speed with speed limit information detected by a camera can determine if a vehicle is speeding. Similarly, based on traffic light color data, timing data, and vehicle status obtained from a camera, it can be determined if a vehicle has run a red light.

[0194] In one implementation, determining the correspondence between objects is necessary when fusing structured violation data. Based on the methods described above for acquiring structured data from lidar and millimeter-wave radar, the coordinate systems of lidar and millimeter-wave radar are the same; therefore, objects with the same coordinates also correspond to the same objects. The correspondence between objects can be determined based on the coordinate data. When establishing a correspondence between the radar and the camera device, the pixel group coordinate data in the video structured data needs to be transformed according to the installation position of the camera device. The correspondence between objects is then determined based on the transformed coordinates.

[0195] Based on the fourth video structured data in S740, the lidar structured data mentioned above, and the millimeter-wave radar structured data, the following is an example of fused structured data:

[0196] [{"Box1":[100,150,200,250],"Category":"Traffic Light","Color":"Red","Pixel Group Coordinates":[(100,150),(100,250),(200,250),(200,150)],"2D Coordinates":[100,150],"Height":600,"Distance":1500},

[0197] {"Box2":[300,350,400,450],"Category":"Vehicle","Color":"White","Pixel Group Coordinates":[(300,350),(300,450),(400,450),(400,350)],"Voltage in Video":100,"2D Coordinates":[200,250],"Height":120,"Orientation Vector":[1,0],"Voltage":25,"Acceleration":2,"Distance":800},

[0198] {"Box3":[500,550,600,650],"Category":"Pedestrian","Pixel Group Coordinates":[(500,550),(500,650),(600,650),(600,550)],"Velocity in Video":2.5,"2D Coordinates":[300,350],"Height":170,"Orientation Vector":[0,1],"Velocity":1.5,"Acceleration":0,"Distance":500},

[0199] {"Box4":[700,750,800,850],"Category":"Sign","Text":"No Parking","Pixel Coordinates":[(700,750),(700,850),(800,850),(800,750)],"2D Coordinates":[400,450],"Height":230,"Distance":1200},

[0200] {"Box5":[900,950,1000,1050],"Category":"Billboard","Pixel Group Coordinates":[(900,950),(900,1050),(1000,1050),(1000,950)],"2D Coordinates":[500,550],"Height":300,"Distance":950},

[0201] {"Box 6":[1100,1150,1200,1250],"Category":"Dark Lane Line","Pixel Group Coordinates":[(1100,1150),(1100,1250),(1200,1250),(1200,1150)]},

[0202] {"Box 7":[1300,1350,1400,1450],"Category":"Fire Hydrant","Pixel Group Coordinates":[(1300,1350),(1300,1450),(1400,1450),(1400,1350)],"2D Coordinates":[1700,1750],"Height":30,"Distance":400,"Distance":16000}

[0203] {"Coordinates":[1700,1750],"Distance":16000}...]

[0204] After coordinate transformation, the object data corresponding to "Box 1" to "Box 7" in the fourth video structured data can be matched with the LiDAR structured data and millimeter-wave radar structured data to summarize the various structured data and obtain the fused reference object data. Taking the object corresponding to "Box 1" as an example, the distance data obtained from the LiDAR structured data and the millimeter-wave radar structured data are consistent, and the coordinate data corresponds to the pixel group coordinate data in the fourth video structured data. Therefore, the fused reference object data for the object corresponding to "Box 1" includes the box position data, category data, color data, and pixel group coordinate data from the fourth video structured data, as well as the coordinate and distance data from the millimeter-wave radar structured data and the LiDAR structured data, and the height data from the LiDAR structured data.

[0205] Taking the object corresponding to "Box 6" as an example, based on the pixel group coordinate data of this object, neither the LiDAR structured data nor the millimeter-wave radar structured data contains matching coordinates for this pixel group. Therefore, the reference data for the object corresponding to "Box 6" in the fused data only includes the fourth video structured data. Similarly, taking the object with ID 7 in the millimeter-wave radar structured data as an example, based on the coordinate data of this object, neither the LiDAR structured data nor the fourth video structured data contains matching pixel group coordinates or box position data for this object. Therefore, the reference data for the object with ID 7 in the fused millimeter-wave radar structured data only includes the millimeter-wave radar structured data.

[0206] S431-3: Determine the type of violation.

[0207] Optionally, violation types can be sorted according to their importance. For example, if both failure to yield to pedestrians and crossing solid lines are determined to exist, the violation types can be sorted by importance: "failure to yield to pedestrians > crossing solid lines," and the final output violation type can be "failure to yield to pedestrians." Alternatively, the first violation type can be output as "failure to yield to pedestrians," and the second violation type as "crossing solid lines."

[0208] In one implementation, to further improve the accuracy of obtaining violation information, different data can be assigned different weights. For example, the priority of reference data can be set from high to low as: LiDAR structured data > millimeter-wave radar structured data > video structured data. For instance, cameras, LiDAR, and millimeter-wave radar can all acquire distance data between vehicles; the distance data acquired by these three sensors may be the same or different. In adverse weather conditions such as rain, snow, fog, or low light at night, it is difficult for cameras to acquire distance data, potentially leading to significant errors. When the distance data in video structured data, LiDAR structured data, and millimeter-wave radar structured data differs, LiDAR structured data can be prioritized as the basis for determining the violation type and extracting violation video. Combining the position and orientation of pedestrians or vehicles with distance data can determine whether a collision has occurred. For another example, while cameras and LiDAR can determine the position of a target vehicle, when determining whether a vehicle has crossed a solid line, the image captured by the camera is limited by the shooting angle, resulting in partial overlap or obstruction between the vehicle and lane lines in the image, which is detrimental to the judgment of violation behavior and violation time. At this point, road environment data can be obtained through camera devices, and the specific three-dimensional position of the target vehicle in the road can be obtained through lidar, or the specific position of the target vehicle can be calculated by obtaining the distance data between the target vehicle and the vehicle itself. By combining the data obtained by lidar and camera devices, the relationship between the target vehicle and the lane lines can be accurately determined, thereby judging whether the target vehicle has crossed the solid line.

[0209] It should be noted that the collection, storage, transmission and processing of all sensor data involved in this application shall be based on legality and comply with the relevant laws and regulations of the country or region where they are located.

[0210] Optionally, when the user's first input includes a description of the violation, sensor data can be selectively acquired and the violation type identified based on this description. For specific identification methods, refer to step S310. For example, when a user issues a voice command, "The white car in front of me committed a violation, record it," the white car in front of the user's vehicle can be identified as the target vehicle. Only the information of the white car and the road in front of the user needs to be acquired. Since the user's voice command includes vehicle color information, the vehicle color in the violation information can be directly determined to be white without image recognition. When a user issues a voice command, "This car ran a red light, record it," only data related to running the red light needs to be acquired, such as the color data of the traffic light, the timing data of the traffic light, and the position and direction of vehicles near the intersection. It is not necessary to acquire distance data or speed data of other vehicles and pedestrians.

[0211] S431-4: Capture video of traffic violations.

[0212] In one implementation, the video footage from the dashcam within a first preset time period preceding the first time period can be used as the basis for determining the traffic violation video. Alternatively, the video footage from the time the violating vehicle first appears in the dashcam to the first time period can be used as the basis for determining the traffic violation video. For example, when another vehicle illegally changes lanes into the lane where the vehicle is located, the time when that vehicle first appears in the dashcam can be used as the start time of the traffic violation video. The user issues a voice command only after the illegal lane change is completed, and the first time period is used as the end time of the traffic violation video. This time period can cover all traffic violations.

[0213] Optionally, the violation video may also include video from a second preset time period following the first time. That is, after the user instructs the start of violation information collection, the video capture time can be extended to fully preserve the driving information of the violating vehicle. The first and second preset times can be determined based on empirical values ​​(e.g., the first and second preset times could be 15 seconds), or they can be specified by the user. For example, the user can issue a voice command, "The vehicle in front just committed a violation; record the video from 20 seconds prior to that," thus capturing the video from the 20 seconds prior to the first time as the violation video.

[0214] In one implementation, to improve the accuracy of capturing traffic violation videos, the initial time can be combined with information acquired by various sensors in the perception system. The start time of the violation is used as the start time of the video, and the end time is used as the end time. For example, if the user's instruction is "The car in front ran a red light, note it," the video can be started by combining the traffic light color information, traffic light timing information, and the speed and direction of the violating vehicle (the car in front) obtained by the camera. This time can be used as the start time of the video. If the violating vehicle stops or turns in the middle of the intersection, the end time of the video can be used when the vehicle's driving state changes. Furthermore, if the violating vehicle continues to drive after crossing the stop line, the end time can be used when the vehicle continues to drive until it crosses the stop line on the opposite side of the intersection.

[0215] For example, when a vehicle's violation lasts too long, to avoid excessive data volume in the violation video, the starting time of the violation video can be determined by combining the information obtained from various sensors in the perception system with the initial time. The video is then captured within a third preset time period after that starting time. For instance, during traffic jams, some vehicles may occupy the emergency lane for extended periods. When capturing violation videos in this scenario, the time when the camera first enters the emergency lane can be used as the starting time of the violation video. Capturing the video within a third preset time period after that starting time proves that the vehicle is not a special vehicle such as a police car or fire truck, and that there was no emergency preventing the vehicle from driving normally. The vehicle's behavior constitutes a violation, without needing to wait for the vehicle to move from the emergency lane to the normal lane before stopping the violation video capture.

[0216] It should be understood that the above methods of capturing various violation videos can be split and recombined, and the embodiments of this application do not limit this.

[0217] In another implementation, the vehicle or cloud can store sensor data for a fixed period of time prior to the initial event, and combine this data with video from the dashcam to obtain traffic violation information for that fixed period of time. For example, this fixed period could be 5 minutes. The method 432 for obtaining traffic violation information based on stored data may include the following steps:

[0218] S432-1: Acquire data from multiple reference objects based on sensor data.

[0219] S432-2: Integrating data from multiple reference points.

[0220] S432-3: Determine the type of violation.

[0221] For example, before obtaining the initial input from the user, the vehicle or the cloud can store the fused structured traffic violation data and the corresponding violation type at each moment. The method for obtaining the structured traffic violation data can be referenced from the method for obtaining real-time traffic violation information above, and will not be repeated here.

[0222] S432-4: Capture video of traffic violations.

[0223] Figure 8 This diagram illustrates a method 800 for capturing traffic violation videos based on initial user input and stored data. Figure 8 As shown, the method 800 may include:

[0224] S810: Determine whether the first input includes a description of the traffic violation.

[0225] Traffic violation descriptions can be user descriptions of other vehicles' traffic violations, including the type of violation. These descriptions can be determined using LLM (Local Management Model). For example, if the user's first input is "A car ran a red light," the violation description would be "ran a red light," indicating the violation type. Alternatively, if the user's first input is "This car was going too fast," the violation description would be "going too fast."

[0226] For example, when the first input includes violation description information, an attempt is made to retrieve the most recent matching violation from memory and execute S820; otherwise, S850 is executed.

[0227] S820: Determine whether the stored data includes a violation that matches the violation description information.

[0228] Using the first moment as the dividing line, the stored data includes structured information of vehicles or cloud storage at each moment prior to the first moment, along with the corresponding violation type. Violations stored in the vehicle or cloud include the violation type and the corresponding moment. Taking illegal occupation of the emergency lane as an example, if this violation continues for a period of time, then the violation type at each moment within that period will include illegal occupation of the emergency lane.

[0229] Based on the fused structured information and traffic violations stored in vehicles or the cloud, it can be determined whether a violation type matches the violation description. For example, if the violation description is "running a red light," and the stored violation types also include running a red light, then the stored data contains a violation that matches that description. If the violation description is "driving too fast," it's inferred that the user intends to express a violation related to speed, and the violation type matching "driving too fast" is speeding. If the stored violation types include speeding, then the stored data contains a violation that matches that description. The degree of matching between the violation description and the violation type can be determined using the cosine similarity algorithm in natural language processing (NLP).

[0230] For example, if the stored data includes a violation that matches the violation description information, S830 is executed; otherwise, S840 is executed.

[0231] S830: The start time of the violation video is the moment when the violation description information and the stored violation behavior are matched.

[0232] For example, if the violation has ended, the moment when the violation description information and the stored violation type match can be the start time of the stored violation, that is, the moment when the violation type that first matches the violation description information appears; if the violation is ongoing, the moment when the violation description information and the stored violation type match can be the moment when the user describes the violation, that is, the first time corresponding to the first input from the user, or it can be the start time of the stored violation.

[0233] S840: The start time of the violation video is the time corresponding to the first preset time before the first time.

[0234] For some traffic violations that are difficult for sensors to detect, such as continuous lane changes or failure to use turn signals as required, where it is difficult to determine a matching violation from the stored data based on the violation description information, the start time of the violation video is set to the time corresponding to a first preset time preceding the first actual time. For example, assuming the first actual time is 14:01:00, the first preset time is 15 seconds, and the start time of the violation video is 14:00:45.

[0235] S850: Determine whether the stored data within the first preset time period before the first time includes violations.

[0236] When the first input does not include violation description information, the nearest forward time period in memory can be retrieved to determine whether the data stored in that time period includes violation behavior. For example, assuming the first time is 14:01:00 and the first preset time is 15 seconds, it can be determined whether there is any violation behavior in the data stored between 14:00:45 and 14:01:00.

[0237] For example, if the data stored within a first preset time period prior to the first time includes a violation, execute S860; otherwise, execute S840. For instance, assuming the first time is 14:01:00 and the first preset time is 15 seconds, if the data stored between 14:00:45 and 14:01:00 includes a violation, execute S860; otherwise, take 14:00:45 as the start time of the violation video.

[0238] S860: The start time of the violation video is taken as the start time of the violation in the stored data.

[0239] The start time of a traffic violation is the moment when a violation type first appears that matches the description of the violation.

[0240] After determining the start time of the violation video based on S830, S840, and S860, S870 can be executed.

[0241] S870: Determine whether the violation is included in the real-time data.

[0242] Using the first moment as the boundary, real-time data includes structured information and corresponding violation types that are fused together from the first moment and every moment acquired after the first moment.

[0243] If the violation is not included in the real-time data, it is determined that the violation has ended and S880 can be executed; otherwise, S890 is executed.

[0244] S880: The time corresponding to the second preset time after the first time is taken as the termination time of the violation video.

[0245] For example, the second preset time can be the same as or different from the first preset time. Taking a second preset time of 15 seconds as an example, if the first time is 14:01:00, then the termination time of the violation video is 14:01:15.

[0246] S890: The time when the violation ends shall be the time when the violation video ends.

[0247] When the real-time data includes the violation, it is determined that the violation is still ongoing. The video is then captured until the moment when no more data containing the violation appears, which is the moment when data that does not include the violation first appears.

[0248] Optionally, after acquiring the traffic violation video, a visual language model (VLM) can be used to output the violation information. For example, by inputting the extracted traffic violation video into a trained visual language model and combining it with a pre-defined prompting algorithm, structured information about the video can be obtained. The fused structured information can include the video structured information obtained using the VLM.

[0249] S440: Feedback on traffic violation information and / or text of traffic violation reports.

[0250] In one implementation, the structured text description information and violation information corresponding to the first input can be fused according to priority, then summarized and output as a complete text description using LLM. The user description can have a higher priority than the violation information obtained by the vehicle. For example, the structured text description information is {"violation license plate":"ABC123","violation vehicle color":"red","violation behavior":"running a red light"}, and the violation information obtained by the vehicle includes {"violation license plate":"ABC125","violation vehicle color":"white","violation behavior":"crossing a solid line"} and other information. The final generated text description could be: "Currently located on the south side of the intersection of xx Road and xx Road, a vehicle with a red color and license plate ABC123 has been recorded as running a red light. The video length is..." "25 seconds. Do you need to change the violation information?" For example, when the structured text description is {}, the vehicle violation information includes {"violation license plate":"ABC125","violation vehicle color":"white","violation behavior":"crossing solid line"} and other information. The final generated text description could be: "Currently located on the south side of the intersection of xx Road and xx Road, a vehicle with a white color and license plate ABC125 has been recorded violating the traffic law by crossing a solid line. The video length is 25 seconds. Do you need to change the violation information?"

[0251] The vehicle can broadcast the traffic violation report text via voice. Alternatively, the violation report text and / or violation information can be displayed to the user via a display device while or after the voice broadcast. For example, the vehicle's infotainment system can be configured with an on / off switch for automatically displaying violation information. This violation information can be automatically displayed to the user after recognizing their intention to summarize the violation information. Figures 9 to 11 Schematic diagrams illustrating some graphical user interfaces of embodiments of this application are shown. In response to clicking, such as... Figure 5 The operation of the setting control 503 shown in (a) can control the display screen to display, such as Figure 9 The display interface shown includes a navigation bar 910 and a content display area 920. The navigation bar 910 includes controls corresponding to functions such as vehicle status, smart assistant, system, and sound. In response to a user clicking the smart assistant control 901, the content display area 920 can display a function description and on / off controls corresponding to the smart assistant, such as a function description and on / off control 902 for automatically collecting traffic violation information. When the on / off control 902 is detected to be in the "on" state, the system can automatically collect and display traffic violation information upon recognizing a user's voice command related to traffic violation information collection. For example, if a user in the cabin issues a voice command, "The vehicle ahead has committed a traffic violation, record it," the system can automatically display information such as... Figure 10 The GUI shown.

[0252] In one implementation, when a user clicks on the violation information collection control 502, the screen can display an image such as... Figure 10 The GUI shown. Figure 10 As shown, this GUI is the display interface after the traffic violation information is collected. This display interface includes a video display box 1001, a list of traffic violation information 1004, and multiple selection controls 1005. The video display box 1001 also includes playback controls, a current playback time icon, a total video length icon, volume controls, a full-screen control, and a control to display more content. The traffic violation video obtained using the above method is the content represented by the video display box 1001, and the duration of this video is consistent with the duration displayed by the total video length icon. For example, if the total length of the video is 11 seconds, and it is currently playing at the 3-second mark.

[0253] The violation information list 1004 includes various violation information and their corresponding details, with each selection control 1005 corresponding to one violation. For example, when displaying violation information, multiple violations can be pre-selected. When a selection control 1005 is selected, a checkmark icon is displayed in the middle; otherwise, only a square is displayed. The selected violations indicate information that will ultimately be saved or submitted.

[0254] Optionally, geographical location and time information can be added to the text description of the voice broadcast or the list of traffic violations 1004 to provide users with the detailed location and specific time when the traffic violation information was recorded.

[0255] Optionally, Figure 10The displayed interface may also include regional prompt information 1006, used to prompt the user with the necessary information required for reporting traffic violations in that region. In one embodiment, a pre-set traffic violation reporting template in the system can be matched with the traffic violation reporting requirements for that region, providing the user with clear prompt information, and allowing pre-selection according to the traffic violation reporting requirements for that region. For example, the pre-set traffic violation reporting template in the system may include information such as the location of the violation, detailed location, time of the violation, type of violation, and license plate number, controlling the vehicle to pre-select this information. Further, combining the violation location (City A) and the traffic violation reporting requirements for City A in the regional prompt information 1006, and pre-selecting the license plate color, the final displayed pre-selected information will be as follows: Figure 10 As shown.

[0256] S450: Determine whether traffic violation information needs to be changed.

[0257] For example, if the violation information needs to be changed, step S460 is executed; otherwise, step S470 is executed.

[0258] For example, the need to modify traffic violation information can be determined based on the user's voice instructions or actions. For instance, if a user's save or report instruction is received within a fourth preset time period after the text of the traffic violation report has finished being read aloud or after the display of the violation information has ended, it is assumed that the violation information does not need to be modified, and step S470 can be executed. Alternatively, if the user still does not give any instruction after the fourth preset time period, it is assumed that the violation information does not need to be modified, and step S470 can be executed.

[0259] S460: Change traffic violation information.

[0260] Users can modify, add, or delete the traffic violation information provided in step S440. All changes to the traffic violation information can be made via voice instructions or manual operation by the user.

[0261] Taking the modification of traffic violation videos as an example, users can trim or extend the traffic violation video displayed in video display box 1001. For example, based on the user's voice command "Move the video start position forward 5 seconds," the user's intention is to adjust the start time of the traffic violation video to 5 seconds before the current time, and the vehicle will be controlled to re-trim the traffic violation video. Alternatively, based on the user's voice command "The video from the 2nd second to the 6th second is enough," the user's intention is to adjust the start time of the traffic violation video to the 2nd second of the current time and the end time to the 6th second of the current time, and the vehicle will be controlled to re-trim the traffic violation video.

[0262] Optionally, users can also manually capture video clips of traffic violations. For example... Figure 10As shown, the display interface includes a video capture progress bar 1002. In one embodiment, the user can use the video capture progress bar 1002 to adjust the start and end times of the violation video. The video capture progress bar 1002 includes two video capture controls 1003, corresponding to the start and end times of the violation video, respectively. In response to the user's sliding operation on the video capture control 1003, the corresponding duration information is displayed, such as... Figure 10 The values ​​"0:02" and "0:06" in the video capture progress bar 1002 indicate that the traffic violation video corresponding to this value is the final traffic violation video.

[0263] In one implementation, users can modify the text description and traffic violation information based on the voice prompts. For example, the voice prompt might say, "We have captured a video of a traffic violation involving vehicle with license plate number ABC124. The violation is running a red light." Users can input modification commands via voice, such as, "Change the license plate number to ABC123 and add the violation information 'driving against traffic.'" Based on the user's modification command, the license plate number can be changed from ABC124 to ABC123, and the violation type "driving against traffic" can be added. The modified violation information can then be announced via voice, such as, "The license plate number has been changed to ABC123, and violation type 2: driving against traffic has been added."

[0264] The violation information obtained from user voice commands, violation videos, and various sensors in the perception system can be displayed in a violation information list 1004 for user confirmation. Based on the pre-selected violation information in the violation information list 1004, the user can modify the violation information using the selection control 1005. For example, the video display box 1001 shows that the vehicle not only ran a red light but also drove against traffic and occupied its own lane, but the pre-selected violation type only showed "Violation Type 1: Running a Red Light". When the user clicks on the selection control 1005 corresponding to "Violation Type 2: Driving Against Traffic", or when the user's voice instruction indicates that they wish to add information related to driving against traffic, the interface showing "Violation Type 2: Driving Against Traffic" as selected can be displayed.

[0265] When the information displayed in the traffic violation information list 1004 is incorrect, it can be modified via voice or manual means. For example, upon receiving the user's voice input "Change the license plate number to ABC123", the license plate number in the traffic violation information list 1004 can be changed from "ABC124" to "ABC123", and the message "Okay, the license plate number has been changed to ABC123" can be announced via voice. Or, in Figure 10 The display interface shown allows for the configuration of a modification control 1007. When a user clicks on the modification control 1007, the vehicle can be redirected to the modification page, which displays the following: Figure 11 The GUI shown. Figure 11As shown, after detecting that the user clicks on the text corresponding to the license plate number, the input keyboard 1101 can be displayed. Responding to the user clicking on different letter or number controls, the corresponding letter or number can be displayed. When the user clicks on the "OK" control 1102, the modified information is saved, and the user can then proceed to... Figure 10 The interface shown displays the modified violation information.

[0266] In addition to the traffic violation information displayed in the GUI above, the method in this embodiment also supports users adding traffic violation information. For example, upon receiving the user's voice command "increase the duration of the violation," based on the duration information represented by the adjusted video capture progress bar 1002, the duration of the violation is determined to be 4 seconds, and the vehicle's display device can be controlled to display the duration information. Alternatively, in Figure 11 The edit page shown allows you to add a new control 1103. When a user clicks on the new control 1103, the input keyboard 1101 can be displayed directly. Responding to different letter or number controls clicked by the user, the corresponding letter or number can be displayed. When a user clicks on the "OK" control 1102, the page will redirect to... Figure 10 The interface shown displays the newly added traffic violation information.

[0267] It should be understood that the above steps S440 to S460 are in Figure 4 The order of steps S440, S450, and S460 is for illustrative purposes only and is not limited in this embodiment. In actual application, the order of S440, S450, and S460 can be adjusted, and the above steps can be performed simultaneously or multiple times. For example, while the vehicle is making a voice announcement in S440, the user can select the traffic violations in the violation information list 1004.

[0268] S470: Determines whether the user wants to report a traffic violation.

[0269] For example, when a user wants to report a traffic violation, S480 is executed; otherwise, S490 is executed.

[0270] For example, it can be determined whether a user wants to report a traffic violation by recognizing the user's voice or detecting the user's click on a control.

[0271] S480: Redirected to the reporting page.

[0272] For example, when a user's voice command "Report directly" is received, the user can be directly redirected to the reporting page corresponding to the location where the violation occurred. Alternatively, in... Figure 10The displayed interface allows you to set up a report control 1009. When a user clicks on the report control 1009, the user can be redirected to the corresponding report mini-program page for City A, and the relevant information will be automatically filled in. Figure 12 This diagram illustrates a GUI representation of a reporting page according to an embodiment of this application. Figure 12 As shown, the necessary information for reporting traffic violations in City A includes the type of violation, evidence of the report (such as the video of the violation in the evidence display box 1202), time of the violation, location of the violation, license plate number, and license plate color. Optional information includes the duration of the violation. Based on the information finally confirmed in steps S410 to S460 above, all information for reporting traffic violations in City A is automatically filled in. Upon receiving the user's voice command "Report Directly" or detecting the user clicking the submit control 1201, the system can... Figure 12 The violation information shown is submitted to the traffic management system of City A.

[0273] In one implementation, at least one violation image can be extracted from the traffic violation video obtained in step S430 at equal intervals according to the regional reporting template, and the extracted violation images can be displayed. When there is only one violation image, the image at the moment the violation occurred can be directly extracted. For example, when a vehicle collision occurs, the image of the frame corresponding to the moment of collision can be selected as the violation image. When the violation lasts for a period of time, such as driving over a solid line for an extended period, any frame during the duration of the violation can be extracted as the violation image.

[0274] When there are multiple violation images, they can be cropped at equal intervals according to the regional reporting template based on the violation video. For example, if a region requires at least two images of the same vehicle to be uploaded when reporting a violation, and the duration of the violation video is 4 seconds, then images from the 0th and 4th seconds can be cropped as violation images. When reporting a violation requires at least three images of the same vehicle, images from the 0th, 2nd, and 4th seconds can be cropped as violation images. The cropped violation images can then be... Figure 12 The evidence display box 1202 in the system displays the violation. Optionally, if the user needs to modify the violation image, they can instruct the vehicle to re-capture the violation video via voice command; or, they can click the delete control 1203 in the upper right corner of the evidence display box 1202. When the user clicks the delete control 1203, the system can automatically delete the violation video. Figure 12 The displayed interface redirects to a manual image capture page, where users can capture images themselves.

[0275] S490: Save traffic violation information.

[0276] For example, if no voice command regarding reporting is received from the user or no click is detected on the corresponding reporting control, and it is determined that the user does not wish to report the violation at this time, the aforementioned violation information can be saved to a folder. For example, the merged structured information, violation type, violation time, etc., can be combined into structured data and stored in memory. This violation information can be stored locally or uploaded to the cloud.

[0277] For example, upon receiving the user's voice command "Save" or detecting the user clicking the save control 1008, the modified violation information can be saved, and the message "Okay, saved" can be announced via voice and / or displayed on the screen as shown above. Figure 13 The GUI shown. Figure 13 As shown, the save page can display specific violation information and a success message, as well as the video storage location for easy retrieval by the user. The saved violation information can be the violation information selected in the selection control 1005 or manually extracted video information, such as the violation location, detailed location, violation time, violation type 1, license plate number, license plate color, and video from second to sixth second; or it can be the original complete violation information, such as including violation type 2, weather, informant information, a summary description of the violation, and a complete violation video with a total duration of 11 seconds.

[0278] Optionally, in the display Figure 10 , Figure 11 or Figure 12 After the displayed interface appears, if the vehicle does not receive any response within a fifth preset time, the violation information can be automatically saved according to the content displayed on the interface to prevent the loss of the acquired violation information. "Not receiving any response" includes the vehicle not receiving any voice commands from the user related to reporting or saving, and not detecting any user interaction with any control on the display screen.

[0279] Optionally, if the location and time information are not obtained and displayed in the aforementioned steps, the location and time information can be incorporated before reporting or saving the violation information.

[0280] After step S480 or S490 is completed, you can return to step S410 for the next collection of violation information.

[0281] Based on the above technical solutions, the method for obtaining traffic violation information provided in this application embodiment can quickly and accurately record traffic violation information, improve the efficiency of obtaining traffic violation information, and help improve user experience.

[0282] Figure 14A schematic block diagram of a traffic violation information acquisition device 1400 provided in an embodiment of this application is shown. The traffic violation information acquisition device 1400 includes: an acquisition unit 1410, configured to acquire a first input from a user, the first input instructing the vehicle to acquire traffic violation information; in response to acquiring the first input, acquiring first data collected by a camera device outside the vehicle's cabin and second data collected by at least one radar; a generation unit 1420, configured to generate a traffic violation report text based on the first data and the second data, the traffic violation report text including multiple pieces of traffic violation information, the multiple pieces of traffic violation information including at least one of the following: violation time, violation location, license plate number, license plate color, and violation type; and a control unit 1430, configured to control the vehicle's prompting device to prompt the user with the traffic violation report text.

[0283] Optionally, the first input can be voice input, which includes violation description information. The acquisition unit 1410 is further configured to: acquire violation description information in response to acquiring the first input; the generation unit 1420 is further configured to generate a violation report text based on the violation description information, the first data, and the second data.

[0284] Optionally, the control unit 1430 is also used to: control the vehicle's display device to display multiple traffic violation information.

[0285] Optionally, the multiple violation information includes first violation data, and the device further includes: a modification unit, configured to modify the first violation data into second violation data in response to receiving input from a user to modify the first violation data.

[0286] Optionally, the first data and the second data are structured data. The generation unit 1420 is specifically used to: match the location data in the first data with the coordinate data in the second data to obtain a matching result; determine the structured violation data based on the matching result, the first data, and the second data; and generate a violation report text based on the structured violation data.

[0287] Optionally, the multiple violation data, including violation types, are used by the generation unit 1420 to: determine the violation type based on the structured violation data; and generate a violation report text based on the violation type, the first data, and the second data.

[0288] Optionally, the acquisition unit 1410 is further configured to acquire the weather conditions of the area where the vehicle is located in response to acquiring the first input; the generation unit 1420 is specifically configured to: when the weather conditions are severe weather or nighttime, generate a violation report text based on at least one of the following data: violation type, first data, and second data: distance data, coordinate data, and speed data.

[0289] Optionally, the violation type is speeding, and the generation unit 1420 is specifically used to generate a violation report text based on the violation type, the category data in the first data, the text data, and the speed data in the second data.

[0290] Optionally, the violation type is solid line, and the generation unit 1420 is specifically used to: generate a violation report text based on the violation type, road environment data and category data in the first data and coordinate data in the second data.

[0291] Optionally, the generation unit 1420 is specifically used to: determine multiple violation information based on the first data, the second data, and the violation reporting template of the area where the vehicle is located.

[0292] Optionally, the acquisition unit 1410 is also used to: acquire a traffic violation video based on the video stored in the dashcam, the time of the first input, the first data, and the second data; and acquire at least one traffic violation image based on the traffic violation reporting template and the traffic violation video.

[0293] Optionally, the device may further include a sending unit for sending multiple violation information in response to detecting a user report of a violation.

[0294] For example, the acquisition unit 1410 can be Figure 1 The computing platform 120 or the processing circuit, processor or controller in the computing platform 120. Taking the processor 121 in the computing platform as an example, the acquisition unit 1410 can acquire a first input from the user, which is used to instruct the vehicle to acquire traffic violation information; in response to acquiring the first input, it can acquire first data collected by the camera device outside the vehicle's cabin and second data collected by at least one radar.

[0295] For example, generating unit 1420 could be Figure 1 The computing platform 120 or the processing circuit, processor, or controller in the computing platform 120. Taking the generation unit 1420 as the processor 122 in the computing platform as an example, the processor 122 can generate a violation report text based on the first data and the second data obtained by the processor 121.

[0296] For example, the control unit 1430 can be Figure 1 The computing platform 120 or the processing circuit, processor, or controller within the computing platform 120. Taking the control unit 1430 as an example, which is the processor 123 in the computing platform, the processor 123 can control the vehicle's prompting device to prompt the user with the violation report text generated by the processor 122.

[0297] For example, modifying a unit could be Figure 1The computing platform 120 or the processing circuit, processor, or controller in the computing platform 120. Taking the processor 124 in the computing platform as an example, the processor 124, in response to receiving the user's input to modify the first violation data, modifies the first violation data to the second violation data.

[0298] For example, the transmitting unit could be Figure 1 The computing platform 120 or the processing circuit, processor, or controller within the computing platform 120. Taking the processor 125 in the computing platform as an example, the processor 125 can send multiple violation information in response to detecting a user's report of a violation.

[0299] The functions implemented by the acquisition unit 1410, generation unit 1420, control unit 1430, modification unit, and sending unit can be implemented by different processors, or by the same processor, or some functions can be implemented by the same processor. This application embodiment does not limit this.

[0300] It should be understood that the division of units in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the units in the device can be implemented by a processor calling software; for example, the device includes a processor connected to memory, which stores instructions. The processor calls the instructions stored in memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be, for example, a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. The functions of some or all units can be implemented through the design of the hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all units are implemented through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby implementing the functions of some or all units. All units of the above devices can be implemented entirely through processor calling software, or entirely through hardware circuits, or partially through processor calling software with the remaining parts implemented through hardware circuits.

[0301] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0302] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0303] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together as a System-on-a-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and AI processor, CPU and GPU, etc.

[0304] This application also provides a traffic violation information acquisition device, which includes: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, so that the device can execute any of the methods described in the above embodiments.

[0305] Alternatively, if the device is located in a vehicle, the processor may be... Figure 1 The processors shown are 121-12n.

[0306] This application also provides a vehicle that may include the aforementioned device 1400.

[0307] This application also provides a computer-readable medium storing instructions that, when executed by a processor, cause the processor to implement any of the methods described in the above embodiments.

[0308] This application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to perform any of the methods described in the above embodiments.

[0309] This application also provides a chip that includes circuitry that can be used to perform any of the methods described in the above embodiments.

[0310] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software units within the processor. The software units can reside in random access memory, flash memory, read-only memory, programmable read-only memory, power-on erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0311] It should be understood that in the embodiments of this application, the memory may include read-only memory and random access memory, and provides instructions and data to the processor.

[0312] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0313] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0314] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0315] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0316] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0317] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0318] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0319] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for obtaining traffic violation information, characterized in that, The method includes: Obtain first input from the user, which instructs the vehicle to obtain traffic violation information; In response to receiving the first input, first data collected by a camera outside the vehicle's cabin and second data collected by at least one radar are acquired. Based on the first data and the second data, a traffic violation report text is generated. The traffic violation report text includes multiple pieces of traffic violation information, including at least one of the following: violation time, violation location, license plate number, license plate color, and violation type. The vehicle's notification device displays the traffic violation report text to the user.

2. The method according to claim 1, characterized in that, The first input is voice input, and the first input includes a description of the traffic violation. The method further includes: In response to receiving the first input, the violation description information is obtained; The step of generating a violation report text based on the first data and the second data includes: The violation report text is generated based on the violation description information, the first data, and the second data.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The vehicle's display device displays the multiple traffic violation information.

4. The method according to any one of claims 1 to 3, characterized in that, The plurality of violation information includes first violation data, and the method further includes: In response to receiving user input to modify the first violation data, the first violation data is modified to the second violation data.

5. The method according to any one of claims 1 to 4, characterized in that, The first data and the second data are structured data. The step of generating a violation report text based on the first data and the second data includes: The position data in the first data is matched with the coordinate data in the second data to obtain the matching result; Based on the matching results, the first data, and the second data, structured violation data is determined; The violation report text is generated based on the structured violation data.

6. The method according to claim 5, characterized in that, The multiple violation information includes the violation type, and the method further includes: Based on the structured violation data, the type of violation is determined; The violation report text is generated based on the violation type, the first data, and the second data.

7. The method according to claim 6, characterized in that, The method further includes: In response to receiving the first input, the weather conditions of the area where the vehicle is located are obtained; The step of generating the violation report text based on the violation type, the first data, and the second data includes: When the weather condition is severe weather or nighttime, the violation report text is generated based on the violation type, the first data, and at least one of the distance data, coordinate data, and speed data in the second data.

8. The method according to claim 6, characterized in that, The violation type is speeding. The step of generating the violation report text based on the violation type, the first data, and the second data includes: The violation report text is generated based on the violation type, the category data and text data in the first data, and the speed data in the second data.

9. The method according to claim 6, characterized in that, The violation type is a solid line. The step of generating the violation report text based on the violation type, the first data, and the second data includes: The violation report text is generated based on the violation type, the road environment data and category data in the first data, and the coordinate data in the second data.

10. The method according to any one of claims 1 to 9, characterized in that, The step of generating a violation report text based on the first data and the second data includes: Based on the first data, the second data, and the violation reporting template for the area where the vehicle is located, the multiple violation information is determined.

11. The method according to claim 10, characterized in that, The multiple violation information includes violation videos, and the method further includes: Based on the video stored in the dashcam, the time of the first input, the first data, and the second data, the traffic violation video is obtained; Based on the traffic violation reporting template and the traffic violation video, obtain at least one traffic violation image.

12. The method according to any one of claims 1-11, characterized in that, The method further includes: In response to detecting a user's report of a traffic violation, the multiple violation information messages are sent.

13. A device for obtaining traffic violation information, characterized in that, The device includes: The acquisition unit is configured to acquire a first input from a user, the first input being used to instruct the vehicle to acquire traffic violation information; in response to acquiring the first input, it acquires first data collected by a camera device outside the vehicle's cabin and second data collected by at least one radar. The generation unit is configured to generate a traffic violation report text based on the first data and the second data. The traffic violation report text includes multiple pieces of traffic violation information, including at least one of the following: violation time, violation location, license plate number, license plate color, and violation type. The control unit is used to control the vehicle's notification device to display the traffic violation report text to the user.

14. The apparatus according to claim 13, characterized in that, The first input is voice input, and the first input includes a description of the traffic violation. The acquisition unit is further configured to acquire the violation description information in response to acquiring the first input; The generation unit is specifically used to generate the violation report text based on the violation description information, the first data, and the second data.

15. The apparatus according to claim 13 or 14, characterized in that, The control unit is also used for: The vehicle's display device displays the multiple traffic violation information.

16. The apparatus according to any one of claims 13 to 15, characterized in that, The plurality of traffic violation information includes first violation data, and the device further includes: The modification unit is configured to modify the first violation data to the second violation data in response to receiving input from the user to modify the first violation data.

17. The apparatus according to any one of claims 13 to 16, characterized in that, The first data and the second data are structured data, and the generation unit is specifically used for: The position data in the first data is matched with the coordinate data in the second data to obtain the matching result; Based on the matching results, the first data, and the second data, structured violation data is determined; The violation report text is generated based on the structured violation data.

18. The apparatus according to claim 17, characterized in that, The plurality of violation information includes the violation type, and the generating unit is further configured to: Based on the structured violation data, the type of violation is determined; The violation report text is generated based on the violation type, the first data, and the second data.

19. The apparatus according to any one of claims 13 to 18, characterized in that, The generation unit is specifically used for: Based on the first data, the second data, and the violation reporting template for the area where the vehicle is located, the multiple violation information is determined.

20. The apparatus according to any one of claims 13-19, characterized in that, The device further includes: The sending unit is used to send the multiple violation information in response to the detection of a user's report of a violation.

21. A device for obtaining traffic violation information, characterized in that, The device includes: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to cause the apparatus to perform the method as described in any one of claims 1 to 12.

22. A vehicle, characterized in that, Includes the apparatus as described in any one of claims 13 to 21.

23. A computer-readable storage medium, characterized in that, It stores instructions that, when executed by a processor, cause the processor to implement the method as described in any one of claims 1 to 12.

24. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 12.

25. A chip, characterized in that, The chip includes circuitry for performing the method as described in any one of claims 1 to 12.

Images