Road surface patrol video generation method and device, road surface patrol vehicle and electronic equipment

Abstract

The invention relates to a pavement patrol video generation method and device, a pavement patrol vehicle and electronic equipment. The method is applied to a road patrol vehicle, camera equipment and positioning equipment are installed on the road patrol vehicle, and network time service is carried out between the camera equipment and the positioning equipment. Acquiring a pavement video through camera equipment, wherein a video frame of the pavement video contains an acquisition timestamp; positioning equipment is used for positioning the road patrol vehicle in real time to obtain positioning information, wherein the positioning information contains a positioning timestamp; determining positioning information corresponding to the acquisition timestamp according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp; and writing the acquisition timestamps and the positioning information corresponding to the acquisition timestamps into meta-information of the video frames with the acquisition timestamps, and generating a road patrol video stream according to each frame of video. A continuous road condition video stream with time information and positioning information can be provided, and the efficiency of rechecking the change condition of the road condition of a road section along with time by a manager is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a road patrol vehicle method, device, road patrol vehicle, electronic equipment, computer-readable storage medium and computer program product. Background Art

[0002] In road maintenance, the common practice of road condition inspection is for inspectors to drive patrol cars for regular inspections and visually check whether there are any abnormalities on the road. This manual inspection method is inefficient and unreliable. To this end, related technologies have proposed a method of using camera equipment to collect images for defect detection, and using image processing technology to analyze images to identify defects in road facilities.

[0003] However, these technologies still have some problems in practical applications. For example, for detected defects, managers need to manually locate and identify the road conditions of a certain section of road frame by frame in a large number of images, and the review efficiency is very low. Summary of the invention

[0004] Based on this, it is necessary to provide a road patrol vehicle method, device, road patrol vehicle, electronic device, computer-readable storage medium and computer program product to address the above technical problems, which can provide a continuous road condition video stream with time information and positioning information, thereby greatly improving the efficiency of management personnel in reviewing the road conditions of a certain section of road and reviewing the changes in the road conditions of the section over time.

[0005] In a first aspect, the present application provides a method for generating a road patrol video, the method being applied to a road patrol vehicle, the road patrol vehicle being equipped with a camera device and a positioning device, comprising:

[0006] Performing network timing between the camera device and the positioning device;

[0007] Collecting road surface video by the camera device, wherein the video frame of the road surface video contains a collection timestamp;

[0008] The road patrol vehicle is positioned in real time by the positioning device to obtain positioning information, wherein the positioning information contains a positioning timestamp;

[0009] Determine the positioning information corresponding to the acquisition timestamp according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp;

[0010] The acquisition timestamp and the positioning information corresponding to the acquisition timestamp are written into the meta information of the video frame with the acquisition timestamp, and a road patrol video stream is generated according to each frame of video.

[0011] In a second aspect, the present application also provides a road patrol video generation device, comprising:

[0012] Timing module, used for network timing between camera equipment and positioning equipment;

[0013] A collection module, used for collecting road surface video through the camera device, wherein the video frame of the road surface video contains a collection timestamp;

[0014] A positioning module, used to obtain positioning information by real-time positioning of the road patrol vehicle through the positioning device, wherein the positioning information contains a positioning timestamp;

[0015] a calculation module, configured to determine the positioning information corresponding to the acquisition timestamp according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp;

[0016] The generating module is used to write the acquisition timestamp and the positioning information corresponding to the acquisition timestamp into the meta information of the video frame with the acquisition timestamp, and generate a road patrol video stream according to each frame of video.

[0017] In some embodiments, there are multiple camera devices, and the timing module is also used to run a server of the time synchronization service on the positioning device; run a client of the time synchronization service on each camera device; and apply for timing from the server on the positioning device using the client through each camera device.

[0018] In some embodiments, the acquisition module is further used to acquire video frames at a first frequency through the camera device; and write the current timestamp as the acquisition timestamp into the meta information of the video frame.

[0019] In some embodiments, the positioning module is also used to obtain the positioning information of the road patrol vehicle by real-time positioning the road patrol vehicle at a second frequency through the positioning device, and determine the positioning timestamp of the positioning information according to the current timestamp, wherein the second frequency is less than the first frequency; and send the positioning information and the positioning timestamp of the positioning information to each of the camera devices through the positioning device.

[0020] In some embodiments, the calculation module is also used to cache the latest positioning information and positioning timestamp through the camera device, and discard the last cached positioning information and positioning timestamp; store the video frames captured by the camera device into a pre-processing image queue; take out the video frames from the pre-processing image queue in sequence, and determine the positioning information corresponding to the acquisition timestamp according to the acquisition timestamp of the video frame, the cached positioning information and the positioning timestamp.

[0021] In some embodiments, the positioning information of the road patrol vehicle includes positioning coordinates, ground speed and true north track direction; the calculation module is further used to determine the time difference based on the acquisition timestamp and the positioning timestamp; calculate the product of the ground speed, the cosine value of the true north track direction and the time difference; calculate the cosine value of the positioning coordinates; and obtain the positioning information corresponding to the acquisition timestamp based on the ratio of the product to the cosine value of the positioning coordinates.

[0022] In some embodiments, the generation module is further used to write the acquisition timestamp and the positioning information corresponding to the acquisition timestamp into the metadata of the video frame with the acquisition timestamp, and store the video frame into a processed image queue; and take out the video frames from the processed image queue in sequence to generate a road patrol video stream.

[0023] In a third aspect, the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0024] Conduct network timing between the camera device and the positioning device;

[0025] Collecting road surface video by the camera device, wherein the video frame of the road surface video contains a collection timestamp;

[0026] The road patrol vehicle is positioned in real time by the positioning device to obtain positioning information, wherein the positioning information contains a positioning timestamp;

[0027] Determine the positioning information corresponding to the acquisition timestamp according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp;

[0028] The acquisition timestamp and the positioning information corresponding to the acquisition timestamp are written into the meta information of the video frame with the acquisition timestamp, and a road patrol video stream is generated according to each frame of video.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0030] Conduct network timing between the camera device and the positioning device;

[0031] Collecting road surface video by the camera device, wherein the video frame of the road surface video contains a collection timestamp;

[0032] The road patrol vehicle is positioned in real time by the positioning device to obtain positioning information, wherein the positioning information contains a positioning timestamp;

[0033] Determine the positioning information corresponding to the acquisition timestamp according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp;

[0034] The acquisition timestamp and the positioning information corresponding to the acquisition timestamp are written into the meta information of the video frame with the acquisition timestamp, and a road patrol video stream is generated according to each frame of video.

[0035] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0036] Conduct network timing between the camera device and the positioning device;

[0037] Collecting road surface video by the camera device, wherein the video frame of the road surface video contains a collection timestamp;

[0038] The road patrol vehicle is positioned in real time by the positioning device to obtain positioning information, wherein the positioning information contains a positioning timestamp;

[0039] Determine the positioning information corresponding to the acquisition timestamp according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp;

[0040] The acquisition timestamp and the positioning information corresponding to the acquisition timestamp are written into the meta information of the video frame with the acquisition timestamp, and a road patrol video stream is generated according to each frame of video.

[0041] In a sixth aspect, the present application further provides a road patrol vehicle, the vehicle being configured with a camera device, a positioning device, a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0042] Conduct network timing between the camera device and the positioning device;

[0043] Collecting road surface video by the camera device, wherein the video frame of the road surface video contains a collection timestamp;

[0044] The road patrol vehicle is positioned in real time by the positioning device to obtain positioning information, wherein the positioning information contains a positioning timestamp;

[0045] Determine the positioning information corresponding to the acquisition timestamp according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp;

[0046] The acquisition timestamp and the positioning information corresponding to the acquisition timestamp are written into the meta information of the video frame with the acquisition timestamp, and a road patrol video stream is generated according to each frame of video.

[0047] The above-mentioned road patrol video generation method, device, electronic device, computer-readable storage medium and computer program product perform network time synchronization between the camera device and the positioning device, so that the clock consistency between the camera device and the positioning device can be achieved, which is the basis for the road condition and road positioning in the subsequently generated video stream to correspond, and is convenient for management personnel to check the changes of road conditions of a certain road section over time. The road video is collected by the camera device, and the video frame of the road video contains a collection timestamp. The positioning information is obtained by real-time positioning of the road patrol vehicle by the positioning device. The positioning information contains a positioning timestamp. According to the collection timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp, the positioning information corresponding to the collection timestamp is determined, and the collection timestamp and the positioning information corresponding to the collection timestamp are written into the meta information of the video frame with the collection timestamp, which can provide a video frame with time information and positioning information, so that the road patrol video stream can be generated according to each frame of video. The management personnel can check the road conditions of a certain road section by watching the road patrol video stream, which greatly improves the efficiency of checking the changes of road conditions of the road section over time. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A diagram showing an application environment of a road patrol video generation method in an embodiment;

[0050] Figure 2 A schematic diagram of a process of generating a road patrol video in one embodiment;

[0051] Figure 3 A schematic diagram of an application architecture of a road patrol video generation method in an embodiment;

[0052] Figure 4 A schematic diagram of a flow chart of a method for generating a road patrol video in another embodiment;

[0053] Figure 5 A schematic diagram of a flow chart of a method for generating a road patrol video in a specific embodiment;

[0054] Figure 6 is a structural block diagram of a road patrol video generating device in one embodiment;

[0055] Figure 7FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] The road patrol video generation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. In some embodiments, the road patrol vehicle 100 includes at least one camera device 101, a positioning device 102, and a central control processing unit 103. In some embodiments, the positioning device 102 can be integrated in the central control processing unit 103, that is, the central control processing unit 103 is a unit integrating timing and positioning, that is, an industrial computer integrating timing and positioning. In some embodiments, there can be multiple camera devices 101, and the multiple cameras are respectively installed at different positions of the road patrol vehicle 100, which can collect video of the road conditions at various positions of the road surface during the movement of the road patrol vehicle 100.

[0058] In some embodiments, both the camera device 101 and the positioning device 102 can apply for network timing from the central control processing unit 103. In some embodiments, the positioning device 102 can be integrated into the central control processing unit 103, so that only the camera device 101 can apply for timing from the central control processing unit 103. Network timing refers to transmitting standard time information to devices that need to be synchronized through the network to achieve time consistency. Due to the influence of the hardware, software and network environment of the camera device 101 and the positioning device 102, their internal clocks are often difficult to keep synchronized. Network timing technology introduces external time sources (such as atomic clocks, GNSS satellites, etc.) and uses time synchronization protocols to transmit standard time information to each node, thereby achieving time synchronization. In the embodiment of the present application, the time information of the camera device 101 and the positioning device 102 can be kept consistent through network timing, especially when there are multiple camera devices 101, time synchronization between multiple camera devices 101 and between the camera device 101 and the positioning device 102 can be achieved, so that synchronized, multi-angle road condition videos can be provided later.

[0059] In some embodiments, the positioning device 102 is used to achieve high-precision GNSS (Global Navigation Satellite System) positioning of road patrol vehicles to obtain positioning information. In some embodiments, the positioning device 102 can provide GPS positioning services at a frequency of not less than 10 Hz. In some embodiments, after completing the network timing, the positioning device 102 can use the current time at the time of positioning as the positioning timestamp to obtain positioning information and the corresponding positioning timestamp. In some embodiments, the positioning device 102 can send the positioning information and the corresponding positioning timestamp to the central control processing unit 103 or the camera device 101 through the local area network.

[0060] In some embodiments, the camera device 101 can capture road video after completing network timing. In some embodiments, the camera device 101 can capture video frames at a frequency of 120 Hz. In some embodiments, the camera device 101 can use the current time when the video frame is captured as the acquisition timestamp, and write the acquisition timestamp into the meta information of the video frame (for example, EXIF ​​information). In some embodiments, the camera device 101 can send the video frame with the acquisition timestamp to the central control processing unit 103.

[0061] In some embodiments, the central control processing unit 103 is used to obtain the video frames collected by the camera device 101, and store the video frames in the pre-processing image queue each time a video frame is collected. In some embodiments, the central control processing unit 103 is used to cache the latest positioning information and the corresponding positioning timestamp each time the latest positioning information and the corresponding positioning timestamp sent by the positioning device 102 are received, and discard the last cached positioning information and positioning timestamp. In some embodiments, the central control processing unit 103 is also used to determine the positioning information corresponding to the acquisition timestamp based on the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp, write the acquisition timestamp and the positioning information corresponding to the acquisition timestamp into the meta information of the video frame with the acquisition timestamp, and generate a road patrol video stream based on each frame of video.

[0062] In some embodiments, the road patrol vehicle 100 is equipped with electronic equipment, which includes a camera, a positioning device, and a central control processing unit. The camera, the positioning device, and the central control processing unit communicate via a local area network.

[0063] In an exemplary embodiment, Figure 2 As shown, a road patrol video generation method is provided, and the method is applied to Figure 1 Taking the electronic device on the road patrol vehicle 100 as an example, the method includes the following steps 202 to 210. Among them:

[0064] Step 202: Perform network timing between the camera device and the positioning device.

[0065] In some embodiments, there are multiple cameras, and network timing is performed between the camera devices and the positioning device, including: running a server of the time synchronization service on the positioning device; running a client of the time synchronization service on each camera device; and applying for timing from the server on the positioning device using the client through each camera device.

[0066] In some embodiments, the positioning device can run the PTP (Precision Time Protocol) server, and the camera device can run the PTP client. The camera device requests time synchronization from the positioning device at a frequency of about 1 Hz. After time synchronization, the time error between multiple camera devices and between the camera device and the positioning device does not exceed 0.1 milliseconds.

[0067] In some embodiments, there are multiple cameras, and network timing is performed between the camera devices and the positioning device, including: running a server of the time synchronization service on the central control processing unit; running a client of the time synchronization service on each camera device and the positioning device respectively; applying for timing from the server on the positioning device by using the client through each camera device, and applying for timing from the server on the positioning device by using the client through the positioning device.

[0068] Step 204: collect road surface video by means of a camera device, wherein the video frame of the road surface video contains a collection timestamp.

[0069] In some embodiments, there are multiple camera devices, which are installed at different positions of the road patrol vehicle to capture road conditions from different perspectives at the same time. For example, some camera devices are used to capture videos of guardrails on the road, some camera devices are used to capture videos of signs on the road, and some cameras are used to capture videos of potholes on the road.

[0070] In some embodiments, after completing network timing, the electronic device may capture video frames at a first frequency through a camera device; and write the current timestamp as the capture timestamp into the meta information of the video frame.

[0071] The electronic device can start the camera device to collect road videos through the camera device. Whenever a video frame is collected, the current time when the video frame is collected is written into the metadata of the video frame as the collection timestamp t1. It can be understood that the collection timestamp is the collection time after the time synchronization is completed, and belongs to the same clock as the positioning timestamp of the positioning device.

[0072] In some embodiments, the electronic device can store the video frames with the acquisition timestamp written into the pre-processing image queue. For example, the three different positions of the road patrol vehicle are respectively according to the camera device C1, the camera device C2 and the camera device C3, and the electronic device can sequentially store the video frames collected by the camera device 1 into the pre-processing image queue Q1, the video frames collected by the camera device 2 into the pre-processing image queue Q2, and the video frames collected by the camera device 3 into the pre-processing image queue Q3.

[0073] Step 206: Use a positioning device to perform real-time positioning of the road patrol vehicle to obtain positioning information, where the positioning information contains a positioning timestamp.

[0074] In some embodiments, the electronic device can obtain the positioning information of the road patrol vehicle by real-time positioning of the road patrol vehicle at a second frequency through a positioning device, and determine the positioning timestamp of the positioning information according to the current timestamp, wherein the second frequency is less than the first frequency.

[0075] Specifically, after completing the time synchronization between the positioning device and the camera device, the positioning device can obtain the positioning information of the road patrol vehicle through the RTK real-time dynamic positioning system at a second frequency not less than 10 Hz. In some embodiments, the second frequency is not less than 10 Hz. In some embodiments, the positioning information includes latitude Lat_0, longitude Lon_0, ground speed v and true north track direction θ. Whenever the positioning device obtains positioning information, the current time is added as the positioning timestamp t0. In some embodiments, the electronic device can send the positioning information and the positioning timestamp of the positioning information to each camera device through the positioning device, for example, it can be sent to the camera device through the TCP network protocol.

[0076] Step 208: Determine the positioning information corresponding to the collection timestamp according to the collection timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp.

[0077] Specifically, when each video frame is collected, the collection timestamp of the video frame can be used to linearly infer the positioning information corresponding to the video frame according to the time difference, using the most recently obtained positioning information as the initial state of motion, that is, the positioning information corresponding to the collection timestamp.

[0078] In some embodiments, the positioning information of the road patrol vehicle includes positioning coordinates, ground speed and true north track direction; based on the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp, the positioning information corresponding to the acquisition timestamp is determined, including: determining the time difference based on the acquisition timestamp and the positioning timestamp; calculating the product of the ground speed, the cosine value of the true north track direction and the time difference; calculating the cosine value of the positioning coordinates; and obtaining the positioning information corresponding to the acquisition timestamp based on the ratio of the product to the cosine value of the positioning coordinates.

[0079] In some embodiments, the latest positioning information and positioning timestamp can be cached by the camera device, and the last cached positioning information and positioning timestamp can be discarded; the video frames captured by the camera device are stored in the pre-processing image queue; the video frames are taken out from the pre-processing image queue in sequence, and the positioning information corresponding to the acquisition timestamp is determined based on the acquisition timestamp of the video frame, the cached positioning information and the positioning timestamp.

[0080] In some embodiments, the positioning coordinates corresponding to the video frame may be calculated according to the following formula:

[0081] ,

[0082] ;

[0083] in, , are the latitude and longitude corresponding to this video frame, , They are the latest latitude and longitude cached respectively; , They are the ground speed (in meters per second) and the true north track direction (in degrees) in the latest cached positioning information; is the acquisition timestamp of this video frame (in seconds), The cache location timestamp (in seconds).

[0084] By caching the most recent positioning information and the corresponding positioning timestamp, and based on the difference between the acquisition timestamp of the video frame and the positioning timestamp, and inferring based on the ground speed and true north track direction on the basis of the most recent positioning coordinates, the positioning coordinates corresponding to the acquisition timestamp of the video frame can be obtained, that is, the positioning coordinates corresponding to the road surface in this video frame.

[0085] Step 210, write the acquisition timestamp and the positioning information corresponding to the acquisition timestamp into the meta information of the video frame with the acquisition timestamp, and generate a road patrol video stream according to each frame of video.

[0086] In some embodiments, the acquisition timestamp and the positioning information corresponding to the acquisition timestamp can be written into the metadata of the video frame with the acquisition timestamp, and the video frame can be stored in a processed image queue; the video frames can be taken out from the processed image queue in sequence to generate a road patrol video stream.

[0087] In some embodiments, a camera device may generate and output a continuous MJPEG video stream based on each video frame in a processed image queue, wherein each video frame contains positioning coordinates and time information.

[0088] The above-mentioned road patrol video generation method performs network time synchronization between the camera device and the positioning device, which can achieve clock consistency between the camera device and the positioning device. This is the basis for the road condition and road positioning in the subsequently generated video stream to correspond, and is convenient for management personnel to check the changes of road conditions of a certain section over time. The road video is collected by the camera device, and the video frame of the road video contains a collection timestamp. The positioning information is obtained by real-time positioning of the road patrol vehicle through the positioning device. The positioning information contains a positioning timestamp. According to the collection timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp, the positioning information corresponding to the collection timestamp is determined, and the collection timestamp and the positioning information corresponding to the collection timestamp are written into the meta information of the video frame with the collection timestamp, which can provide a video frame with time information and positioning information, so that a road patrol video stream can be generated according to each frame of video. The management personnel can check the road conditions of a certain section by watching the road patrol video stream, which greatly improves the efficiency of checking the changes of road conditions of the section over time.

[0089] Figure 3 FIG. 1 is a schematic diagram of an application architecture of a method for generating a road patrol video in an embodiment. Figure 3 The application architecture includes multiple camera devices 301, a network switch 302, an integrated timing and positioning industrial computer 303, and a GNSS antenna 304. Each camera device 301 is connected to the network switch 302 via a local area network, and the integrated timing and positioning industrial computer 303 is connected to the network switch 302 via a local area network, so that the camera device 301 can communicate with the integrated timing and positioning industrial computer 303. The GNSS antenna 304 obtains satellite signals and transmits them to the integrated timing and positioning industrial computer 303, so as to perform GPS positioning on the road patrol vehicle through the positioning system.

[0090] Figure 4 FIG. 1 is a flow chart of a method for generating a road patrol video in one embodiment. Figure 4, the client running the time synchronization protocol on the camera device, and the server running the time synchronization protocol on the integrated positioning industrial computer. First, the client on the camera device applies for network timing from the server on the integrated timing and positioning industrial computer through the time synchronization protocol. After the timing is completed, the camera device collects road video at a frequency of about 120Hz, and each video frame is added with a collection timestamp. After the timing is completed, the integrated timing and positioning industrial computer obtains the latitude and longitude, ground speed, direction and other positioning information of the current road patrol vehicle through the GNSS positioning system at a frequency of not less than 10Hz, adds the current time as the positioning timestamp, and sends the positioning information and positioning timestamp to the camera device. Then, the camera device calculates the positioning information corresponding to the video frame according to the acquisition timestamp of the video frame and the latest received positioning information and positioning timestamp, and writes the acquisition timestamp and positioning information into the metadata of the video frame. Finally, the camera device can generate a continuous video stream according to the acquisition timestamp sequence of the video frame and output the road patrol video stream.

[0091] Figure 5 FIG. 1 is a flow chart of a method for generating a road patrol video in a specific embodiment. Figure 5 , including five steps: multi-camera timing (501), obtaining positioning information (502), image acquisition (503), image corresponding positioning information generation (504), and video stream and related positioning information output (505). Among them:

[0092] Multi-camera timing: The server of the time synchronization protocol of the integrated timing and positioning industrial computer, and the client of the camera device running the time synchronization protocol; the camera device applies for timing from the industrial computer at a frequency of about 1Hz. After timing, the time error between multiple camera devices and between the camera device and the integrated timing and positioning industrial computer shall not exceed 0.1 milliseconds.

[0093] Obtaining positioning information: The integrated timing and positioning industrial computer obtains the latitude of the patrol car at a frequency of no less than 10Hz through the RTK real-time dynamic positioning system. ,longitude , ground speed , True North Track Direction ; Every time the timing and positioning integrated industrial computer obtains positioning information, it adds the current timestamp as the positioning timestamp The camera device listens through the TCP protocol and receives positioning information every time , the last positioning information is discarded and the latest received positioning information and positioning timestamp are cached.

[0094] Image acquisition: After the timing is completed, the camera device acquires high-definition video images at a frame rate of about 120fps, and each time a frame is acquired, the current time is used as the acquisition timestamp. Write the EXIF ​​information of this frame of image and save the video frame with the acquisition timestamp to the pre-processing image queue .

[0095] Image corresponding positioning information generation: The camera device performs the following processing on the pre-processed image queue frame by frame. According to the calculation formula, based on the acquisition timestamp of the video frame, the latest cached positioning information and positioning timestamp, the positioning information corresponding to the video frame is calculated, the positioning information and acquisition timestamp are written into the EXIF ​​information of this frame, and this frame of image is saved to the processed image queue .

[0096] Video stream and related positioning information output: The camera device queues the processed images. For each video frame in the image, a continuous MJPEG video stream is generated and output, in which each frame contains coordinate information and time information.

[0097] In some embodiments, the camera equipment is installed outside the road patrol vehicle to shoot the road surface, guardrails, signs, etc., and the timing and positioning integrated industrial computer is installed on the top of the road patrol vehicle or inside the vehicle as needed.

[0098] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a road patrol video generation device for implementing the road patrol video generation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the road patrol video generation device provided below can refer to the limitations of the road patrol video generation method above, and will not be repeated here.

[0100] In an exemplary embodiment, Figure 6 As shown, a road patrol video generation device 600 is provided, including: a timing module 602, a collection module 604, a positioning module 606, a calculation module 608 and a generation module 610, wherein:

[0101] The timing module 602 is used to perform network timing between the camera device and the positioning device;

[0102] The acquisition module 604 is used to acquire road surface video through a camera device, and the video frame of the road surface video contains an acquisition timestamp;

[0103] The positioning module 606 is used to obtain positioning information by real-time positioning of the road patrol vehicle through the positioning device, and the positioning information contains a positioning timestamp;

[0104] A calculation module 608, configured to determine the positioning information corresponding to the collection timestamp according to the collection timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp;

[0105] The generating module 610 is used to write the acquisition timestamp and the positioning information corresponding to the acquisition timestamp into the meta information of the video frame with the acquisition timestamp, and generate a road patrol video stream according to each frame of video.

[0106] In some embodiments, there are multiple cameras, and the timing module 602 is also used to run the server of the time synchronization service on the positioning device; run the client of the time synchronization service on each camera respectively; and apply for timing from the server on the positioning device through each camera using the client.

[0107] In some embodiments, the acquisition module 604 is further configured to acquire video frames at a first frequency through a camera device; and write the current timestamp as the acquisition timestamp into the meta information of the video frame.

[0108] In some embodiments, the positioning module 606 is also used to obtain the positioning information of the road patrol vehicle by real-time positioning of the road patrol vehicle at a second frequency through a positioning device, and determine the positioning timestamp of the positioning information according to the current timestamp, wherein the second frequency is less than the first frequency; and send the positioning information and the positioning timestamp of the positioning information to each camera device through the positioning device.

[0109] In some embodiments, the calculation module 608 is also used to cache the latest positioning information and positioning timestamp through the camera device, and discard the last cached positioning information and positioning timestamp; store the video frames captured by the camera device into the pre-processing image queue; take out the video frames from the pre-processing image queue in sequence, and determine the positioning information corresponding to the acquisition timestamp according to the acquisition timestamp of the video frame, the cached positioning information and the positioning timestamp.

[0110] In some embodiments, the positioning information of the road patrol vehicle includes positioning coordinates, ground speed and true north track direction; the calculation module 608 is also used to determine the time difference based on the acquisition timestamp and the positioning timestamp; calculate the product of the ground speed, the cosine value of the true north track direction and the time difference; calculate the cosine value of the positioning coordinates; and obtain the positioning information corresponding to the acquisition timestamp based on the ratio of the product to the cosine value of the positioning coordinates.

[0111] In some embodiments, the generation module 610 is further used to write the acquisition timestamp and the positioning information corresponding to the acquisition timestamp into the metadata of the video frame with the acquisition timestamp, store the video frame into the processed image queue; take out the video frames from the processed image queue in sequence to generate a road patrol video stream.

[0112] The above-mentioned road patrol video generation device 600 performs network time synchronization between the camera device and the positioning device, so that the clocks between the camera device and the positioning device can be consistent. This is the basis for the road condition and road positioning in the subsequently generated video stream to correspond, which is convenient for management personnel to check the changes of road conditions of a certain road section over time. The road video is collected by the camera device, and the video frame of the road video contains a collection timestamp. The positioning information is obtained by real-time positioning of the road patrol vehicle through the positioning device. The positioning information contains a positioning timestamp. According to the collection timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp, the positioning information corresponding to the collection timestamp is determined, and the collection timestamp and the positioning information corresponding to the collection timestamp are written into the meta information of the video frame with the collection timestamp, which can provide a video frame with time information and positioning information, so that a road patrol video stream can be generated according to each frame of video. The management personnel can check the road conditions of a certain road section by watching the road patrol video stream, which greatly improves the efficiency of checking the changes of road conditions of the road section over time.

[0113] Each module in the road patrol video generation device 600 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0114] In an exemplary embodiment, an electronic device is provided, which can be used as an electronic device for road patrol on a road patrol vehicle, and its internal structure diagram can be as follows: Figure 7As shown. The electronic device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external device through a network connection. When the computer program is executed by the processor, a road patrol video generation method is implemented.

[0115] Those skilled in the art will understand that Figure 7 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0116] In an exemplary embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of a road patrol video generation method when executing the computer program.

[0117] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the road patrol video generation method are implemented.

[0118] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the road patrol video generation method when executed by a processor.

[0119] In one embodiment, a road patrol vehicle is provided, which is equipped with a positioning device, a camera device memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the road patrol video production method provided in the above embodiment are implemented.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0121] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0122] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0123] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating a road patrol video, characterized in that: The method is applied to a road patrol vehicle, on which a camera device and a positioning device are installed, and the method comprises: Performing network timing between the camera device and the positioning device; Collecting road surface video by the camera device, wherein the video frame of the road surface video contains a collection timestamp; The road patrol vehicle is positioned in real time by the positioning device to obtain positioning information, wherein the positioning information contains a positioning timestamp; Determine the positioning information corresponding to the acquisition timestamp according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp; The acquisition timestamp and the positioning information corresponding to the acquisition timestamp are written into the meta information of the video frame with the acquisition timestamp, and a road patrol video stream is generated according to each frame of video.

2. The method according to claim 1, characterized in that There are multiple camera devices, and network timing is performed between the camera devices and the positioning device, including: A server running a time synchronization service on the positioning device; Run the client of the time synchronization service on each of the camera devices respectively; Each of the camera devices uses a client to apply for time synchronization from a server on the positioning device.

3. The method according to claim 1, characterized in that: The collecting of road surface video by the camera device includes: Capturing video frames at a first frequency by the camera device; The current timestamp is written as the acquisition timestamp into the meta information of the video frame.

4. The method according to claim 3, characterized in that The real-time positioning of the road patrol vehicle by the positioning device to obtain positioning information includes: The positioning device performs real-time positioning of the road patrol vehicle at a second frequency to obtain positioning information of the road patrol vehicle, and determines a positioning timestamp of the positioning information according to a current timestamp, wherein the second frequency is less than the first frequency; The positioning information and the positioning timestamp of the positioning information are respectively sent to each of the camera devices through the positioning device.

5. The method according to claim 1, characterized in that The method further comprises: caching the latest positioning information and positioning timestamp by the camera device, and discarding the last cached positioning information and positioning timestamp; Storing the video frames collected by the camera device into a pre-processing image queue; The determining, according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp, the positioning information corresponding to the acquisition timestamp includes: The video frames are sequentially taken out from the pre-processed image queue, and the positioning information corresponding to the acquisition timestamp is determined according to the acquisition timestamp of the video frame, the cached positioning information and the positioning timestamp.

6. The method according to any one of claims 1 to 5, characterized in that: The positioning information of the road patrol vehicle includes positioning coordinates, ground speed and true north track direction; The determining, according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp, the positioning information corresponding to the acquisition timestamp includes: Determine a time difference according to the acquisition timestamp and the positioning timestamp; Calculating the product of the ground speed, the cosine value of the true north track direction, and the time difference; Calculating the cosine value of the positioning coordinates; The positioning information corresponding to the acquisition timestamp is obtained according to the ratio of the product to the cosine value of the positioning coordinates.

7. The method according to any one of claims 1 to 5, characterized in that: The step of writing the acquisition timestamp and the positioning information corresponding to the acquisition timestamp into the meta information of the video frame having the acquisition timestamp, and generating a road patrol video stream according to each frame of video, comprises: Writing the acquisition timestamp and the positioning information corresponding to the acquisition timestamp into the meta information of the video frame with the acquisition timestamp, and storing the video frame into a processed image queue; Video frames are sequentially taken from the processed image queue to generate a road patrol video stream.

8. A road patrol video generation device, characterized in that: The device comprises: Timing module, used for network timing between camera equipment and positioning equipment; A collection module, used for collecting road surface video through the camera device, wherein the video frame of the road surface video contains a collection timestamp; A positioning module, used to obtain positioning information by real-time positioning of the road patrol vehicle through the positioning device, wherein the positioning information contains a positioning timestamp; a calculation module, configured to determine the positioning information corresponding to the acquisition timestamp according to the acquisition timestamp, the positioning timestamp and the positioning information corresponding to the positioning timestamp; The generating module is used to write the acquisition timestamp and the positioning information corresponding to the acquisition timestamp into the meta information of the video frame with the acquisition timestamp, and generate a road patrol video stream according to each frame of video.

9. A road patrol vehicle, characterized in that: The vehicle is equipped with a camera device, a positioning device, a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images