Automatic violation report processing method, system and device and electronic equipment
By performing voice recognition and image processing on the user side, and generating and sending traffic violation reporting information, the problems of cumbersome reporting methods and insufficient accuracy in the existing technology are solved, and efficient and accurate traffic violation reporting is achieved.
Patent Information
- Application Number
- CN202411608564.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-12
- Publication Date
- 2025-02-18
AI Technical Summary
In the existing technology, the methods of reporting traffic violations are cumbersome, the accuracy and reliability are insufficient, and the real-time nature is lacking, so it is impossible to report the violation as soon as it occurs.
By realizing voice recognition and image processing on the user's side, the voice information input by the user is obtained and the violation description content is extracted, and the violation report information is generated based on the traffic images captured by the target camera, and it is sent to the violation processing server for processing.
It realizes automatic violation reporting without manual participation by users, improves the efficiency and accuracy of traffic violation reporting, and ensures timely handling of violations.
Smart Images
Figure CN119479323A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent transportation technology, and in particular to an automatic violation reporting and processing method, system, device and electronic equipment. Background Art
[0002] With the development of intelligent transportation systems, road safety management has become an important research direction. Among them, reporting road violation information is an important part of road safety management, which can help traffic management departments to timely discover and deal with road violations and improve road safety.
[0003] In the prior art, the main way to report road violation information is for users to manually record and submit information. For example, you can manually enter information such as license plate number and violation type through a mobile phone or a dedicated reporting platform, and then submit it to the traffic management department. In addition, there are some technologies that automatically detect and record road violation information through vehicle-mounted equipment, such as a driving recorder, and then upload it to the traffic management department.
[0004] However, there are some problems with the existing methods of reporting road violations. First, the manual reporting method is cumbersome and requires road users to spend a certain amount of time and energy, which is a burden for drivers. Secondly, although the automatic detection and reporting method of on-board equipment can reduce human intervention, its accuracy and reliability need to be improved, and false detection or missed detection may occur. Finally, both methods lack real-time performance and cannot report violations as soon as they occur. Summary of the invention
[0005] In view of this, in order to solve some or all of the above-mentioned technical problems, the embodiments of the present application provide a method, system, device, electronic device, storage medium and vehicle for automatically reporting and processing traffic violations.
[0006] In a first aspect, an embodiment of the present application provides an automatic violation reporting processing method, which is applied to a user end, and the method includes: in response to triggering a violation reporting instruction, obtaining voice information input by a user; recognizing the voice information to obtain a violation description content; based on the violation description content, extracting a traffic image taken by a target camera; based on the violation description content, the traffic image and the voice information, generating violation reporting information; and sending the violation reporting information to a violation processing server, so that the violation processing server generates reporting result information based on the violation reporting information.
[0007] In one possible implementation, voice information is recognized to obtain and extract violation description content, including: converting voice information into text information; performing named entity recognition on the text information to obtain entity information used to describe the violation phenomenon of the reported object; and generating violation description content based on the entity information.
[0008] In one possible implementation, named entity recognition is performed on text information to obtain entity information used to describe the violation of the reported object, including: named entity recognition is performed on text information to obtain at least one of the following entity information: location entity information, license plate number entity information, violation type entity information, and appearance entity information of the reported object.
[0009] In a possible implementation, extracting a traffic image taken by a target camera based on the violation description content includes: determining a traffic image taken by a target camera corresponding to the location entity information based on the location entity information included in the violation description content.
[0010] In a possible implementation, after sending the traffic violation reporting information to the traffic violation processing server, the method further includes: receiving reporting result information sent by the traffic violation processing server, and outputting the reporting result information in a preset output manner.
[0011] In a second aspect, an embodiment of the present application provides a method for processing traffic violation reporting information, which is applied to a server, and the method includes: receiving traffic violation reporting information sent by a user terminal, and extracting traffic violation description content, traffic images and voice information from the traffic violation reporting information, wherein the traffic violation reporting information is generated by the user terminal in the following manner: in response to triggering a traffic violation reporting instruction, obtaining voice information input by the user; recognizing the voice information to obtain extracted traffic violation description content; based on the traffic violation description content, extracting a traffic image taken by a target camera; based on the traffic violation description content, traffic images and voice information, generating traffic violation reporting information; based on the traffic violation description content and / or voice information, performing target detection and traffic violation behavior detection on the traffic image to obtain the location information of the reported object and the corresponding violation confidence; if the violation confidence is greater than or equal to a preset confidence threshold, generating reporting result information indicating that the reported object has committed a traffic violation; and sending the reporting result information to the user terminal.
[0012] In one possible implementation, after performing target detection and violation behavior detection on traffic images based on violation description content and / or voice information to obtain location information of the reported object and the corresponding violation confidence, the method further includes: if the violation confidence is less than the confidence threshold, sending the violation report information to the manual review end; receiving the report result information sent by the manual review end, and sending the report result information to the user end.
[0013] In a third aspect, an embodiment of the present application provides an automatic violation reporting processing system, which includes: a user terminal and a violation processing server; the user terminal is used to execute the automatic violation reporting processing method described in the first aspect above; the violation processing server is used to execute the violation reporting information processing method described in the second aspect above.
[0014] In a fourth aspect, an embodiment of the present application provides an automatic violation reporting processing device, which is applied to a user end, and the device includes: an acquisition module, which is used to obtain voice information input by a user in response to triggering a violation reporting instruction; a recognition module, which is used to recognize the voice information and obtain the violation description content; a second extraction module, which is used to extract the traffic image taken by the target camera based on the violation description content; a first generation module, which is used to generate violation reporting information based on the violation description content, traffic image and voice information; a first sending module, which is used to send the violation reporting information to a violation processing server, so that the violation processing server generates reporting result information based on the violation reporting information.
[0015] In a fifth aspect, an embodiment of the present application provides a violation reporting information processing device, which is applied to a server, and the device includes: a receiving module, which is used to receive the violation reporting information sent by the user terminal, and extract the violation description content, traffic image and voice information from the violation reporting information, wherein the violation reporting information is generated by the user terminal in the following manner: in response to triggering a violation reporting instruction, obtaining the voice information input by the user; recognizing the voice information to obtain the extracted violation description content; based on the violation description content, extracting the traffic image taken by the target camera; generating the violation reporting information based on the violation description content, traffic image and voice information; a detection module, which is used to perform target detection and violation behavior detection on the traffic image based on the violation description content and / or voice information, and obtain the location information of the reported object and the corresponding violation confidence; a second generation module, which is used to generate a reporting result information indicating that the reported object has committed a violation if the violation confidence is greater than or equal to a preset confidence threshold; and a second sending module, which is used to send the reporting result information to the user terminal.
[0016] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and when the computer program is executed, it implements the method of any embodiment of the automatic violation reporting processing method described in the first aspect or the second aspect of the present application.
[0017] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method of any embodiment of the automatic violation reporting processing method described in the first aspect or the second aspect above is implemented.
[0018] In an eighth aspect, an embodiment of the present application provides a computer program, which includes a computer readable code. When the computer readable code runs on a device, the processor in the device implements a method as in any embodiment of the method for automatic violation reporting processing in the first aspect mentioned above.
[0019] In the ninth aspect, an embodiment of the present application provides a vehicle, including: a user-end device, a microphone, and a camera; the microphone is used to collect the user's voice information; the camera is used to capture the traffic image of the road where the vehicle is located; the user-end device is used to execute the automatic violation reporting processing method described in the first aspect above based on the voice information and traffic images.
[0020] The automatic traffic violation reporting processing method, device, electronic device and storage medium provided in the embodiment of the present application, when the traffic violation reporting instruction is triggered, recognizes the voice information input by the user to obtain the description of the traffic violation, extracts the traffic image captured by the target camera, generates the traffic violation reporting information based on the description of the traffic violation, the traffic image and the voice information, and sends the traffic violation reporting information to the traffic violation processing server. The embodiment of the present application enables the user to automatically generate the traffic violation reporting information by combining voice and image, and the traffic violation reporting information is processed by the traffic violation processing server, without the need for the user to manually participate in the generation process of the traffic violation reporting information, thereby improving the efficiency of traffic violation reporting, and the captured traffic image corresponds to the description of the traffic violation described by the user, thereby improving the accuracy of traffic violation behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0023] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.
[0024] Figure 1 A flowchart of an automatic violation reporting method provided in an embodiment of the present application;
[0025] Figure 2 A flowchart of another automatic violation reporting method provided in an embodiment of the present application;
[0026] Figure 3 A flowchart of a method for processing traffic violation reporting information provided in an embodiment of the present application;
[0027] Figure 4A flowchart of another method for processing traffic violation reporting information provided in an embodiment of the present application;
[0028] Figure 5 A schematic diagram of the structure of an automatic violation reporting and processing system provided in an embodiment of the present application;
[0029] Figure 6 A schematic diagram of the structure of an automatic violation reporting and processing device provided in an embodiment of the present application;
[0030] Figure 7 A schematic diagram of the structure of a violation reporting information processing device provided in an embodiment of the present application;
[0031] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that unless otherwise specifically stated, the relative arrangement of the components and steps, the numerical expressions and the numerical values described in these embodiments do not limit the scope of the present application.
[0033] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present application are only used to distinguish between different steps, devices, modules and other objects, and do not represent any specific technical meanings, nor do they indicate the logical order between them.
[0034] It should also be understood that in this embodiment, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.
[0035] It should also be understood that any component, data or structure mentioned in the embodiments of the present application can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.
[0036] In addition, the term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.
[0037] It should also be understood that the description of the various embodiments in this application focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.
[0038] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.
[0039] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the above-mentioned technologies, methods, and equipment should be considered as part of the specification.
[0040] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0041] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. To facilitate the understanding of the embodiments of the present application, the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0042] In order to solve the technical problems of low efficiency and low accuracy in reporting traffic violations in the prior art, the present application provides an automatic violation reporting processing method, which can automatically generate violation reporting information based on user voice and images, thereby improving the efficiency of traffic violation reporting and the accuracy of violation identification.
[0043] Figure 1 A flow chart of an automatic violation reporting method provided in an embodiment of the present application. The method can be applied to one or more electronic devices such as a controller, a smart phone, a laptop, a portable computer, etc. on a vehicle. These electronic devices can be referred to as user terminals, and the user terminals can implement a voice recognition function and an image capture function. The electronic device executing the method is connected to a violation processing server for communication, and the generated violation reporting information is sent to the violation processing server in real time.
[0044] In addition, the execution subject of the present method may be hardware or software. When the execution subject is hardware, the execution subject may be one or more of the electronic devices. For example, a single electronic device may execute the present method, or a plurality of electronic devices may cooperate with each other to execute the present method. When the execution subject is software, the present method may be implemented as a plurality of software or software modules, or as a single software or software module. No specific limitation is made herein.
[0045] like Figure 1 As shown, the method specifically includes:
[0046] Step 101, in response to triggering a traffic violation reporting instruction, obtaining voice information input by a user.
[0047] In this embodiment, the electronic device executing this method can respond to the traffic violation reporting instruction triggered by the user's traffic violation reporting operation under the user's operation. For example, the electronic device can be a controller on a vehicle, and the controller can collect the user's voice instructions through the microphone on the vehicle, and when the controller recognizes the user's voice signal reporting other vehicles' traffic violations, the traffic violation reporting instruction is triggered. For another example, when the user presses a button set with the traffic violation reporting function, the traffic violation reporting instruction is triggered.
[0048] After the traffic violation reporting command is triggered, the electronic device can obtain the user's voice information collected by the microphone.
[0049] Step 102, recognize the voice information to obtain the violation description content.
[0050] In this embodiment, the electronic device may be provided with a voice recognition function so that voice information can be recognized.
[0051] The electronic device can perform semantic recognition on the voice information and extract the content used to describe the violation of the reported object. The reported object can be various types of objects, for example, vehicles (can be motor vehicles or non-motor vehicles), pedestrians, etc. The electronic device can use natural language recognition methods (for example, using a pre-trained natural language model) to perform semantic recognition on the voice information, extract preset types of keywords from the voice information, and use these keywords as the description of the violation. As an example, information such as the license plate number, vehicle model, and location where the violation occurred of the reported object can be extracted from the text information.
[0052] Step 103: extracting the traffic image captured by the target camera based on the violation description content.
[0053] In this embodiment, the target camera can be a camera included in the electronic device executing the method, or a camera connected to the electronic device. The violation description content is related to the shooting area of the target camera, and the target camera corresponding to the corresponding area can be determined according to the violation description content.
[0054] For example, the user's vehicle includes multiple cameras, and the violation description may include the positional relationship between the reported object and the vehicle (for example, the violation description includes directional words such as "in front", "left", and "behind" that indicate the positional relationship). Based on the positional relationship, the camera that photographed the reported object can be determined to be the target camera.
[0055] The above traffic image may be a single image or a sequence of multiple images. When the traffic violation reporting instruction is triggered, the traffic image taken at the current moment may be extracted, or the traffic images taken continuously for a period of time before and after the current moment may be extracted.
[0056] Step 104, generating violation report information based on the violation description content, traffic image and voice information.
[0057] In this embodiment, the violation description content, traffic image and voice information can be directly combined as the violation report information. Other information (such as reporting time, reporting location, etc.) can also be added to the combined content to obtain the violation report information.
[0058] Step 105: Send the traffic violation report information to the traffic violation processing server, so that the traffic violation processing server generates report result information based on the traffic violation report information.
[0059] In this embodiment, the traffic violation processing server is a server for receiving traffic violation reporting information, and the server is usually connected to a relevant department that handles traffic violation behaviors to confirm the traffic violation phenomena reported by users.
[0060] The automatic traffic violation reporting processing method provided in the embodiment of the present application, when the traffic violation reporting instruction is triggered, recognizes the voice information input by the user to obtain the description of the traffic violation, extracts the traffic image captured by the target camera, generates the traffic violation reporting information based on the description of the traffic violation, the traffic image and the voice information, and sends the traffic violation reporting information to the traffic violation processing server. The embodiment of the present application enables the user to automatically generate the traffic violation reporting information by combining voice and image, and the traffic violation reporting information is processed by the traffic violation processing server, without the need for the user to manually participate in the generation process of the traffic violation reporting information, thereby improving the efficiency of traffic violation reporting, and the captured traffic image corresponds to the description of the traffic violation described by the user, thereby improving the accuracy of traffic violation behavior recognition.
[0061] In some optional implementations of this embodiment, such as Figure 2 As shown, step 102 includes:
[0062] Step 1021, converting the voice information into text information.
[0063] Specifically, the electronic device may be provided with a speech recognition function so as to recognize the speech information. For example, a speech recognition model may be used to convert the speech information into text information.
[0064] Step 1022, perform named entity recognition on the text information to obtain entity information used to describe the violation of the reported object.
[0065] Specifically, the electronic device may be provided with a pre-trained named entity recognition model, and by inputting text information into the named entity model, multiple types of named entities may be output, and entity information describing the reported object's violation may be obtained according to the pre-set named entity types representing the violation. The entity information describing the reported object's violation may represent features such as the location, license plate number, and violation type of the reported object.
[0066] Step 1023, generating violation description content based on the entity information.
[0067] Specifically, the above entity information can be directly used as the violation description content. Other information (such as entity type and other information) can also be added to the entity information to obtain the violation description content.
[0068] This embodiment can quickly and accurately extract information indicating traffic violations from the voice uttered by the user by performing named entity recognition on text information, thereby improving the efficiency and accuracy of reporting traffic violations.
[0069] In some optional implementations of this embodiment, step 1022 includes:
[0070] Perform named entity recognition on the text information to obtain at least one of the following entity information: location entity information, license plate number entity information, violation type entity information, and appearance entity information of the reported object.
[0071] The above-mentioned location entity information indicates the relative position relationship between the reported object and the user, for example, "in front", "behind", "left", etc.
[0072] The above license plate number entity information represents the license plate number of the reported object (ie, the reported vehicle).
[0073] The above-mentioned violation type entity information indicates the specific violation type of the reported object, such as "illegal lane change", "speeding", "running a red light", etc.
[0074] The above-mentioned appearance entity information of the reported object indicates the appearance characteristics of the reported object. For example, when the reported object is a vehicle, the appearance entity information of the reported object may be "SUV", "pickup truck", "sedan", etc. When the reported object is a pedestrian, the appearance entity information of the reported object may be "young male", "drunk state", etc.
[0075] At least one of the above-mentioned various entity information can be used as information for describing the traffic violation behavior. The specific type of entity information required as information for describing the traffic violation behavior can be preset.
[0076] This embodiment achieves the goal of fully extracting the content related to the violation from the user's voice by setting multiple types of entity information representing the violation phenomenon, so that the violation description content can more comprehensively represent the violation behavior of the reported object, thereby improving the accuracy of the violation report.
[0077] In some optional implementations of this embodiment, step 104 includes:
[0078] Based on the position entity information included in the violation description content, a traffic image captured by a target camera corresponding to the position entity information is determined.
[0079] Specifically, the orientation entity information represents the relative position relationship between the reported object and the user, and the camera on the user's vehicle or the reporting tool used has a certain correspondence with the identified orientation. Therefore, based on the orientation entity information, the target camera that can photograph the reported object can be determined from multiple cameras, and then the traffic image taken by the target camera can be obtained, and it can be determined that the traffic image contains the image of the reported object when the violation occurs.
[0080] In some optional implementations of this embodiment, after step 104, the method further includes:
[0081] Receive the report result information sent by the violation processing server and output the report result information according to the preset output method.
[0082] Among them, the violation processing server can further process the violation reporting information to determine whether the reported object has actually committed a violation and the specific type of violation. The generated reporting result information can indicate whether the reported object has actually committed a violation and the specific type of violation.
[0083] After receiving the report result information, the electronic device may automatically output the report result information according to the output method selected by the user. For example, the report result information may be output in the form of voice broadcast or in the form of text display on the vehicle screen.
[0084] This embodiment can enable users to know the reporting results in time by outputting the reporting result information sent by the violation processing server, thereby improving the execution efficiency of the reporting process.
[0085] Figure 3 A flowchart of a method for processing traffic violation reporting information provided in an embodiment of the present application. The method can be applied to a traffic violation processing server. The traffic violation processing server can receive the traffic violation reporting information sent by the above-mentioned user terminal, execute the method based on the traffic violation reporting information, generate reporting result information, and feed back the reporting result information to the user terminal.
[0086] In addition, the execution subject of the present method may be hardware or software. When the execution subject is hardware, the execution subject may be one or more of the electronic devices. For example, a single electronic device may execute the present method, or a plurality of electronic devices may cooperate with each other to execute the present method. When the execution subject is software, the present method may be implemented as a plurality of software or software modules, or as a single software or software module. No specific limitation is made herein.
[0087] like Figure 3 As shown, the method specifically includes:
[0088] Step 301, receiving traffic violation report information sent by a user terminal, and extracting traffic violation description content, traffic images and voice information from the traffic violation report information.
[0089] In this embodiment, the traffic violation report information is generated by the user end in the following manner: in response to triggering the traffic violation report instruction, the voice information input by the user is obtained; the voice information is recognized to obtain the traffic violation description content; based on the traffic violation description content, the traffic image taken by the target camera is extracted; based on the traffic violation description content, the traffic image and the voice information, the traffic violation report information is generated. For the content of the traffic violation report information, reference can be made to the above embodiment and will not be repeated here.
[0090] Step 302: Based on the violation description content and / or voice information, target detection and violation behavior detection are performed on the traffic image to obtain the location information of the reported object and the corresponding violation confidence.
[0091] Specifically, a pre-trained target detection model can be used to detect the positions of objects such as vehicles and pedestrians from traffic images, as well as the positions and states of objects such as lane lines, traffic signs, and traffic lights. Combined with the above-mentioned violation description content and / or voice information, the position information of the reported object can be determined from the detected vehicles, pedestrians, and other objects. For example, if the violation description content or the voice signal is re-identified and it is determined that the reported object is the vehicle in front of the user, and the license plate number of the vehicle can be obtained, then the position of the vehicle corresponding to the license plate number can be determined from the detected vehicles as the position of the reported object.
[0092] After determining the location of the reported object, the reported object can be further detected by combining the information of detected lane lines, traffic signs and other objects, and using the pre-trained violation detection model to detect the reported object's violation behavior. Specifically, the reported object's behavior can be classified as a violation and the violation confidence can be obtained.
[0093] The traffic violation confidence can represent the probability of a certain type of traffic violation that the reported object has committed. For example, if the type of traffic violation that the reported object has committed is red light running, the confidence of the red light running behavior can be output as the traffic violation confidence.
[0094] The above-mentioned target detection model and violation behavior detection model can be obtained by pre-training the initial model through machine learning methods. That is, a large number of training samples are collected in advance, and the training samples include sample images and the positions and violation types of the labeled violation objects. During training, the sample images are used as the input information of the initial target detection model and the initial violation behavior detection model, and the labeled position information and violation type are used as the expected output information of the model. By minimizing the loss value of the loss function (such as the L1 loss representing the error between the labeled position and the predicted position, the cross entropy loss representing the error between the labeled category and the predicted category, etc.) as the optimization target, the gradient descent method and the back propagation method are used to adjust the parameters of the initial model. When the training end conditions are met, the training is stopped to obtain the trained target detection model and violation behavior detection model.
[0095] Optionally, a pre-trained multimodal target detection model can also be used to extract features from violation description content and / or voice information, as well as from traffic images, and the various features obtained are fused. The location prediction score and violation type prediction branches included in the model are used to predict the location information of the reported object and the violation confidence corresponding to the violation type.
[0096] Step 303: If the traffic violation confidence is greater than or equal to a preset confidence threshold, report result information indicating that the reported object has committed a traffic violation is generated.
[0097] Specifically, the reporting result information may be generated based on the type of traffic violation detected, information of the reported object (eg, license plate number, etc.), reporting time, and other information.
[0098] Optionally, the report result information can be directly used as the result of determining the violation; or, the report result information can also be used as a reference for the relevant departments that handle traffic violations to handle the violation. The electronic device that executes this method can send the report result information to the above-mentioned department, and the above-mentioned department will further determine the violation.
[0099] Step 304, sending the reporting result information to the user end.
[0100] The automatic violation reporting and processing method provided in the embodiment of the present application receives the violation reporting information sent by the user terminal through the violation processing server, performs target detection and violation behavior detection based on the violation reporting information, obtains the location information of the reported object and the corresponding violation confidence, generates reporting result information according to the violation confidence, and sends the reporting result information to the user terminal. The violation reporting information is automatically generated by combining voice and image, and the violation reporting information is processed by the violation processing server, without the need for the user to manually participate in the generation process of the violation reporting information, thereby improving the efficiency of traffic violation reporting, and the traffic image captured corresponds to the violation description content described by the user, thereby improving the accuracy of traffic violation behavior identification.
[0101] In some optional implementations of this embodiment, such as Figure 4 As shown, after step 302, the method further includes:
[0102] Step 305: If the traffic violation confidence is less than the confidence threshold, the traffic violation report information is sent to the manual review end.
[0103] Among them, the manual review end can manually review the violation report information to accurately determine whether the reported object has committed a violation. After the manual review, the report result information is generated and returned to the violation processing server.
[0104] Step 306, receiving the report result information sent by the manual review end, and sending the report result information to the user end.
[0105] The violation processing server will feed back accurate reporting result information to the above-mentioned user end, so that the user end can know whether the report is successful in a timely manner.
[0106] This embodiment helps to more accurately determine the violation by manually reviewing whether the reported object has violated the traffic rules when the violation confidence is less than the confidence threshold.
[0107] Figure 5 The structure diagram of an automatic violation reporting and processing system provided in the embodiment of the present application includes: a user terminal 501 and a violation processing server 502.
[0108] The user terminal 501 is used to execute the above-mentioned automatic violation reporting processing method. The violation processing server 502 is used for the above-mentioned violation reporting information processing method. The user terminal can be various types of terminal devices, such as a controller on a vehicle, or a mobile terminal such as a smart phone. Figure 5 As shown, the user end can be a controller on the vehicle, or a terminal device such as a smart phone connected to the vehicle. The user end is connected to the violation processing server through a network.
[0109] The execution process of the system includes:
[0110] In response to triggering a traffic violation reporting instruction, the user terminal 501 obtains voice information input by the user; recognizes the voice information to obtain a description of the traffic violation; based on the description of the traffic violation, extracts the traffic image taken by the target camera; based on the description of the traffic violation, the traffic image and the voice information, generates the traffic violation reporting information; and sends the traffic violation reporting information to the traffic violation processing server.
[0111] The violation processing server 502 receives the violation reporting information sent by the user terminal, and extracts the violation description content, traffic image and voice information from the violation reporting information; based on the violation description content and / or voice information, performs target detection and violation behavior detection on the traffic image to obtain the location information of the reported object and the corresponding violation confidence; if the violation confidence is greater than or equal to the preset confidence threshold, generates reporting result information indicating that the reported object has committed a violation; and sends the reporting result information to the user terminal.
[0112] The automatic traffic violation reporting processing system provided in the embodiment of the present application realizes automatic generation of traffic violation reporting information through a combination of voice and image by respectively executing corresponding automatic traffic violation reporting processing methods by the user terminal and the traffic violation processing server. The traffic violation reporting information is processed by the traffic violation processing server without the need for users to manually participate in the generation process of the traffic violation reporting information, thereby improving the efficiency of traffic violation reporting. The captured traffic images correspond to the violation description content described by the user, thereby improving the accuracy of traffic violation behavior identification.
[0113] Figure 6 This is a schematic diagram of the structure of an automatic violation reporting and processing device provided in an embodiment of the present application. Specifically comprising:
[0114] An acquisition module 601 is used to acquire voice information input by a user in response to triggering a traffic violation reporting instruction;
[0115] Recognition module 602, used to recognize voice information and obtain the description of the violation;
[0116] An extraction module 603 is used to extract the traffic image captured by the target camera based on the violation description content;
[0117] A first generating module 604, for generating traffic violation reporting information based on the traffic violation description content, traffic images and voice information;
[0118] The first sending module 605 is used to send the traffic violation report information to the traffic violation processing server, so that the traffic violation processing server generates report result information based on the traffic violation report information.
[0119] In one possible implementation, the recognition module includes: a conversion unit for converting voice information into text information; a recognition unit for performing named entity recognition on the text information to obtain entity information for describing the violation of the reported object; and a generation unit for generating violation description content based on the entity information.
[0120] In one possible implementation, the recognition unit is further used to: perform named entity recognition on the text information to obtain at least one of the following entity information: location entity information, license plate number entity information, violation type entity information, and reported object appearance entity information.
[0121] In a possible implementation manner, the second extraction module is further used to: determine, based on the location entity information included in the violation description content, a traffic image captured by a target camera corresponding to the location entity information.
[0122] In a possible implementation manner, the device further includes: an output module, configured to receive the reporting result information sent by the violation processing server, and output the reporting result information in a preset output manner.
[0123] The automatic violation reporting and processing device provided in this embodiment can be as follows Figure 6 The automatic traffic violation reporting processing device shown in can execute all the steps of the above automatic traffic violation reporting processing methods, and thus achieve the technical effects of the above automatic traffic violation reporting processing methods. Please refer to the above related description for details. For the sake of brevity, it will not be repeated here.
[0124] Figure 7 A schematic diagram of a traffic violation reporting information processing device provided in an embodiment of the present application. Specifically comprising:
[0125] The receiving module 701 is used to receive the traffic violation report information sent by the user terminal, and extract the traffic violation description content, traffic image and voice information from the traffic violation report information, wherein the traffic violation report information is generated by the user terminal in the following manner: in response to triggering the traffic violation report instruction, obtaining the voice information input by the user; recognizing the voice information to obtain the extracted traffic violation description content; based on the traffic violation description content, extracting the traffic image taken by the target camera; generating the traffic violation report information based on the traffic violation description content, traffic image and voice information;
[0126] The detection module 702 is used to perform target detection and violation behavior detection on the traffic image based on the violation description content and / or voice information, and obtain the location information of the reported object and the corresponding violation confidence;
[0127] The second generating module 703 is used to generate reporting result information indicating that the reported object has committed a traffic violation if the traffic violation confidence is greater than or equal to a preset confidence threshold;
[0128] The second sending module 704 is used to send the reporting result information to the user end.
[0129] In one possible embodiment, the device also includes: a third sending module, used to send the violation report information to the manual review end if the violation confidence is less than the confidence threshold; a fourth sending module, used to receive the report result information sent by the manual review end, and send the report result information to the user end.
[0130] The violation reporting information processing device provided in this embodiment can be as follows Figure 7 The traffic violation reporting information processing device shown in can execute all the steps of the above automatic traffic violation reporting processing methods, and thus achieve the technical effects of the above traffic violation reporting information processing methods. Please refer to the above related description for details. For the sake of brevity, it will not be repeated here.
[0131] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 8 The electronic device 800 shown includes: at least one processor 801, a memory 802, at least one network interface 804 and other user interfaces 803. The various components in the electronic device 800 are coupled together via a bus system 805. It is understood that the bus system 805 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 805 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 805 is not described in detail. Figure 8 Various buses are labeled as bus system 805 .
[0132] The user interface 803 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).
[0133] It can be understood that the memory 802 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 802 described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0134] In some implementations, the memory 802 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 8021 and an application program 8022 .
[0135] The operating system 8021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 8022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present application can be included in the application 8022.
[0136] In this embodiment, by calling the program or instruction stored in the memory 802, specifically, the program or instruction stored in the application 8022, the processor 801 is used to execute the method steps provided by each method embodiment, for example, including:
[0137] In response to triggering a traffic violation reporting instruction, voice information input by a user is obtained; the voice information is recognized to obtain a description of the traffic violation; based on the description of the traffic violation, a traffic image taken by a target camera is extracted; based on the description of the traffic violation, the traffic image and the voice information, traffic violation reporting information is generated; the traffic violation reporting information is sent to a traffic violation processing server, so that the traffic violation processing server generates reporting result information based on the traffic violation reporting information.
[0138] Or include:
[0139] Receive the traffic violation report information sent by the user end, and extract the traffic violation description content, traffic image and voice information from the traffic violation report information, wherein the traffic violation report information is generated by the user end in the following manner: in response to triggering a traffic violation report instruction, obtain the voice information input by the user; recognize the voice information to obtain the extracted traffic violation description content; based on the traffic violation description content, extract the traffic image taken by the target camera; based on the traffic violation description content, traffic image and voice information, generate the traffic violation report information; based on the traffic violation description content and / or voice information, perform target detection and traffic violation behavior detection on the traffic image to obtain the location information of the reported object and the corresponding violation confidence; if the violation confidence is greater than or equal to a preset confidence threshold, generate the report result information indicating that the reported object has committed a traffic violation; send the report result information to the user end.
[0140] The method disclosed in the above embodiment of the present application can be applied to the processor 801, or implemented by the processor 801. The processor 801 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 801. The above processor 801 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 802, and the processor 801 reads the information in the memory 802 and completes the steps of the above method in combination with its hardware.
[0141] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the above functions of the present application, or a combination thereof.
[0142] For software implementation, the technology described above in this article can be implemented by a unit that performs the functions described above in this article. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.
[0143] The electronic device provided in this embodiment may be Figure 8 The electronic device shown in can execute all the steps of the above-mentioned automatic violation reporting processing methods or violation reporting information processing methods, and thus achieve the technical effects of the above-mentioned automatic violation reporting processing methods or violation reporting information processing methods. Please refer to the above related description for details. For the sake of brevity, it will not be repeated here.
[0144] The embodiment of the present application also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.
[0145] When one or more programs in the storage medium can be executed by one or more processors, the above method executed on the electronic device side is implemented.
[0146] The processor is used to execute the program stored in the memory to implement the following steps of the automatic violation reporting processing method executed on the user side:
[0147] In response to triggering a traffic violation reporting instruction, voice information input by a user is obtained; the voice information is recognized to obtain a description of the traffic violation; based on the description of the traffic violation, a traffic image taken by a target camera is extracted; based on the description of the traffic violation, the traffic image and the voice information, traffic violation reporting information is generated; the traffic violation reporting information is sent to a traffic violation processing server, so that the traffic violation processing server generates reporting result information based on the traffic violation reporting information.
[0148] Or implement the following steps of the violation reporting information processing method executed on the server side:
[0149] Receive the traffic violation report information sent by the user end, and extract the traffic violation description content, traffic image and voice information from the traffic violation report information, wherein the traffic violation report information is generated by the user end in the following manner: in response to triggering a traffic violation report instruction, obtain the voice information input by the user; recognize the voice information to obtain the extracted traffic violation description content; based on the traffic violation description content, extract the traffic image taken by the target camera; based on the traffic violation description content, traffic image and voice information, generate the traffic violation report information; based on the traffic violation description content and / or voice information, perform target detection and traffic violation behavior detection on the traffic image to obtain the location information of the reported object and the corresponding violation confidence; if the violation confidence is greater than or equal to a preset confidence threshold, generate the report result information indicating that the reported object has committed a traffic violation; send the report result information to the user end.
[0150] An embodiment of the present application also provides a vehicle, including: a user terminal device, a microphone, and a camera.
[0151] The microphone is used to collect the user's voice information. The camera is used to capture the traffic image of the road where the vehicle is located. The user terminal device is used to execute the automatic violation reporting processing method described in the above embodiment based on the voice information and the traffic image. The user terminal device can be a controller on the vehicle, or other devices connected to the vehicle and located on the vehicle (such as a smart phone, a tablet computer, etc.).
[0152] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0153] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0154] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments, and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.
[0155] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.
Claims
1. An automatic violation reporting processing method, applied to a user end, characterized in that: The method comprises: In response to triggering a traffic violation reporting instruction, obtaining voice information input by a user; Recognize the voice information to extract the violation description content; Based on the violation description content, extracting a traffic image captured by a target camera; Generate traffic violation reporting information based on the traffic violation description content, the traffic image and the voice information; The traffic violation reporting information is sent to a traffic violation processing server, so that the traffic violation processing server generates reporting result information based on the traffic violation reporting information.
2. The method according to claim 1, characterized in that The step of identifying the voice information to extract the violation description content includes: Converting the voice information into text information; Performing named entity recognition on the text information to obtain entity information used to describe the violation of the reported object; Based on the entity information, the violation description content is generated.
3. The method according to claim 2, characterized in that The performing of named entity recognition on the text information to obtain entity information used to describe the violation of the reported object includes: Perform named entity recognition on the text information to obtain at least one of the following entity information: location entity information, license plate number entity information, violation type entity information, and appearance entity information of the reported object.
4. The method according to claim 3, characterized in that The step of extracting a traffic image captured by a target camera based on the violation description content includes: Based on the position entity information included in the violation description content, a traffic image captured by a target camera corresponding to the position entity information is determined.
5. The method according to claim 1, characterized in that After sending the traffic violation reporting information to the traffic violation processing server, the method further includes: Receive the report result information sent by the violation processing server, and output the report result information according to a preset output method.
6. A method for processing traffic violation reporting information, applied to a server, characterized in that: The method comprises: Receive the traffic violation report information sent by the user terminal, and extract the traffic violation description content, traffic image and voice information from the traffic violation report information, wherein the traffic violation report information is generated by the user terminal in the following manner: in response to triggering a traffic violation report instruction, obtain the voice information input by the user; recognize the voice information to obtain the extracted traffic violation description content; based on the traffic violation description content, extract the traffic image taken by the target camera; generate the traffic violation report information based on the traffic violation description content, the traffic image and the voice information; Based on the violation description content and / or the voice information, target detection and violation behavior detection are performed on the traffic image to obtain location information of the reported object and the corresponding violation confidence; If the violation confidence is greater than or equal to a preset confidence threshold, generating reporting result information indicating that the reported object has committed a violation; The reporting result information is sent to the user terminal.
7. The method according to claim 6, characterized in that After performing target detection and violation behavior detection on the traffic image based on the violation description content and / or the voice information to obtain the location information of the reported object and the corresponding violation confidence, the method further includes: If the violation confidence is less than the confidence threshold, the violation report information is sent to a manual review terminal; Receive the report result information sent by the manual review end, and send the report result information to the user end.
8. An automatic violation reporting and processing system, characterized in that: The system comprises: a user terminal and a violation processing server; The user terminal is used to execute the automatic violation reporting processing method according to any one of claims 1 to 5; The violation processing server is used to execute the violation reporting information processing method described in claim 6 or 7.
9. An automatic violation reporting and processing device, applied to a user end, characterized in that: The device comprises: An acquisition module, used for acquiring voice information input by a user in response to triggering a traffic violation reporting instruction; A recognition module, used to recognize the voice information and obtain the description of the violation; An extraction module, used to extract a traffic image taken by a target camera based on the violation description content; A first generating module, used for generating traffic violation reporting information based on the traffic violation description content, the traffic image and the voice information; The first sending module is used to send the traffic violation reporting information to the traffic violation processing server, so that the traffic violation processing server generates reporting result information based on the traffic violation reporting information.
10. A traffic violation reporting information processing device, applied to a server, characterized in that: The device comprises: A receiving module is used to receive the traffic violation report information sent by the user terminal, and extract the traffic violation description content, traffic image and voice information from the traffic violation report information, wherein the traffic violation report information is generated by the user terminal in the following manner: in response to triggering a traffic violation report instruction, obtaining voice information input by the user; identifying the voice information to obtain the extracted traffic violation description content; based on the traffic violation description content, extracting the traffic image taken by the target camera; generating the traffic violation report information based on the traffic violation description content, the traffic image and the voice information; A detection module, configured to perform target detection and violation behavior detection on the traffic image based on the violation description content and / or the voice information, and obtain the location information of the reported object and the corresponding violation confidence; A second generating module is used to generate reporting result information indicating that the reported object has committed a traffic violation if the traffic violation confidence is greater than or equal to a preset confidence threshold; The second sending module is used to send the reporting result information to the user terminal.
11. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program stored in the memory, and when the computer program is executed, it implements the automatic violation reporting processing method described in any one of claims 1-5 or the violation reporting information processing method described in any one of claims 6-7.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the automatic violation reporting processing method described in any one of claims 1 to 5 or the violation reporting information processing method described in any one of claims 6 to 7 is implemented.
13. A vehicle, characterized in that: Includes: user-end equipment, microphone, camera; The microphone is used to collect the user's voice information; The camera is used to capture traffic images of the road where the vehicle is located; The user terminal device is used to execute the automatic violation reporting processing method described in any one of claims 1 to 5 based on the voice information and the traffic image.