Road condition monitoring method and device, terminal and computer readable storage medium

Abstract

The invention belongs to the technical field of intelligent traffic, and mainly provides a road condition monitoring method and device, a terminal and a computer readable storage medium, and the method comprises the steps: obtaining a vehicle front image collected by a first vehicle, determining a visible farthest position in front of the first vehicle based on the vehicle front image, then obtaining a starting point position with the farthest position as the starting point position, and carrying out the detection of the vehicle front image. A plurality of real-time images are collected by a second vehicle located in a preset distance range in front of a first vehicle, the plurality of real-time images collected by the second vehicle are spliced to obtain a panoramic road condition image which takes a farthest position as a starting point position and is located in the preset distance range in front of the first vehicle, and the panoramic road condition image is displayed by the first vehicle. Therefore, the driver of the first vehicle can predict the road condition in front of the vehicle in advance, the problem that the driver cannot know the specific road condition in front of the vehicle in time in the driving process is solved, traffic jam can be solved, traffic accidents can be prevented, and traffic safety can be improved.

Description

Technical Field

[0001] The present application belongs to the field of intelligent transportation technology, and in particular, relates to a road condition monitoring method, device, terminal and computer-readable storage medium. Background Art

[0002] As people's living standards improve, vehicles are becoming more and more popular. While the popularity of vehicles brings convenience to people's lives, it also brings some traffic problems such as traffic congestion and traffic accidents. Solving traffic congestion, preventing traffic accidents and improving traffic safety are important goals of traffic information system design.

[0003] However, the existing traffic information system mainly relies on fixed traffic cameras installed on the road, but such cameras are rarely directly open to the public. Therefore, currently drivers are usually unable to obtain timely information about the specific road conditions ahead of the vehicle while driving, which is not conducive to solving traffic congestion, preventing traffic accidents and improving traffic safety. Summary of the invention

[0004] The present application provides a road condition monitoring method, device, terminal and computer-readable storage medium, which can reduce the situation where the driver cannot timely know the specific road conditions ahead of the vehicle during driving, which is conducive to solving traffic congestion, preventing traffic accidents and improving traffic safety.

[0005] A first aspect of an embodiment of the present application provides a road condition monitoring method, which is applied to a first vehicle or a cloud server, and includes:

[0006] Acquire a vehicle front image captured by the first vehicle, and determine the farthest visible position in front of the first vehicle based on the vehicle front image;

[0007] Acquire multiple real-time images collected by a second vehicle located within a preset distance range in front of the first vehicle with the farthest position as a starting point;

[0008] The multiple real-time images collected by the second vehicle are stitched together to obtain a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, and the first vehicle displays the panoramic road condition image.

[0009] A second aspect of the embodiment of the present application further provides a road condition monitoring device, which is configured in a first vehicle or a cloud server, and includes:

[0010] A first acquisition unit, configured to acquire a vehicle front image collected by the first vehicle, and determine a farthest visible position in front of the first vehicle based on the vehicle front image;

[0011] A second acquisition unit is used to acquire a plurality of real-time images collected by a second vehicle located within a preset distance range in front of the first vehicle with the farthest position as a starting point;

[0012] A stitching unit is used to stitch multiple real-time images collected by the second vehicle to obtain a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, and the first vehicle displays the panoramic road condition image.

[0013] A third aspect of an embodiment of the present application provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the road condition monitoring method described in the first aspect above.

[0014] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the road condition monitoring method described in the first aspect are implemented.

[0015] In an embodiment of the present application, an image in front of the vehicle captured by the first vehicle is obtained, and the farthest visible position in front of the first vehicle is determined based on the image in front of the vehicle. Then, multiple real-time images captured by a second vehicle with the farthest position as the starting point and located within a preset distance range in front of the first vehicle are obtained, and the multiple real-time images captured by the second vehicle are spliced ​​to obtain a panoramic road condition image with the farthest position as the starting point and located within the preset distance range in front of the first vehicle. The panoramic road condition image is displayed by the first vehicle, so that the driver of the first vehicle can predict the road condition in front of his vehicle in advance, which solves the problem that the driver usually cannot promptly know the specific road condition in front of the vehicle during driving, which is conducive to solving traffic congestion, preventing traffic accidents and improving traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the first implementation flow of the road condition monitoring method provided in an embodiment of the present application.

[0017] Figure 2 A schematic diagram of a preset distance range in front of a first vehicle provided in an embodiment of the present application.

[0018] Figure 3 A schematic diagram of a method for determining a real-time image containing road conditions outside a preset distance range in front of a first vehicle provided in an embodiment of the present application.

[0019] Figure 4 A schematic diagram of a second implementation flow of the road condition monitoring method provided in an embodiment of the present application.

[0020] Figure 5 A schematic diagram of the structure of a road condition monitoring device provided in an embodiment of the present application.

[0021] Figure 6 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] It should be understood that in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0024] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0026] As urban traffic becomes increasingly complex, traffic congestion and frequent accidents have become urgent problems to be solved. Existing traffic information systems mainly rely on fixed traffic cameras installed on the road, but these cameras are rarely directly open to the public. Therefore, drivers are usually unable to obtain the specific road conditions ahead of the vehicle in a timely manner while driving, which is not conducive to solving traffic congestion, preventing traffic accidents and improving traffic safety.

[0027] Based on this, the embodiments of the present application provide a road condition monitoring method, device, terminal and computer-readable storage medium, which can reduce the situation where the driver cannot promptly know the specific road conditions ahead of the vehicle during driving, which is conducive to solving traffic congestion, preventing traffic accidents and improving traffic safety.

[0028] like Figure 1As shown, a schematic diagram of the implementation process of a road condition monitoring method provided in an embodiment of the present application is provided. The road condition monitoring method can be executed by a road condition monitoring device on a terminal, and the terminal can be a first vehicle or a cloud server. The present application does not limit the specific type of the terminal. The road condition monitoring method provided in an embodiment of the present application can include the following steps 101 to 103.

[0029] Step 101 : acquiring a vehicle front image captured by a first vehicle, and determining the farthest visible position in front of the first vehicle based on the vehicle front image.

[0030] In the embodiment of the present application, the first vehicle may be equipped with one or more front cameras, which may be used to capture the road image in front of the first vehicle (vehicle front image). The image content in the vehicle front image can represent the content in the driver's front field of view of the first vehicle. Therefore, the farthest position visible in front of the first vehicle determined based on the vehicle front image is the farthest position visible in front of the driver of the first vehicle.

[0031] In one embodiment of the present application, the above-mentioned image in front of the vehicle can be input into a first preset image recognition model, and the first preset image recognition model outputs the farthest visible position in front of the first vehicle.

[0032] In one embodiment of the present application, a large number of sample images in front of the vehicle can be collected first, and the farthest visible position corresponding to each image in front of the vehicle can be annotated to obtain the farthest visible position label corresponding to each image in front of the vehicle, and then the sample images in front of the vehicle and their corresponding farthest visible position labels can be used to train the pre-constructed first initial model to obtain the above-mentioned first preset image recognition model. Among them, the above-mentioned pre-constructed first initial model can be a convolutional neural network model, and the present application does not limit the type of model.

[0033] Step 102 , obtaining a plurality of real-time images collected by a second vehicle located within a preset distance range in front of the first vehicle with the farthest position as the starting point.

[0034] In the embodiment of the present application, the second vehicle may be configured with a front, rear or side camera for capturing real-time images or videos of the front, rear and sides of the vehicle.

[0035] Specifically, after the second vehicle is started, it can collect image data of the vehicle's current environment and vehicle location information, and send the image data of the vehicle's current environment, vehicle location information, and image data collection timestamp to other vehicles (for example, the first vehicle mentioned above) or roadside equipment (for example, smart traffic lights, roadside cameras and other roadside equipment) through vehicle-to-vehicle interoperability technology (V2X), so that the roadside equipment can send it to other vehicles.

[0036] Step 103, stitching the multiple real-time images collected by the second vehicle to obtain a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, and the first vehicle displays the panoramic road condition image.

[0037] In the embodiment of the present application, since the road range from the current position of the first vehicle to the above-mentioned farthest position is within the field of vision of the driver of the first vehicle, and the road outside the preset distance range in front of the first vehicle is relatively far away and does not need to be concerned about, it is only necessary to obtain multiple real-time images collected by the second vehicle within the preset distance range in front of the first vehicle with the above-mentioned farthest position as the starting point, and splice them, without the need to splice the road images within the field of vision of the driver of the first vehicle or outside the preset distance range in front of the first vehicle, so as to reduce the calculation amount of road condition monitoring.

[0038] It should be noted that, since the application ultimately obtains a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, and the farthest position as the starting point, this section of road located within the preset distance range in front of the first vehicle is likely to contain a blind spot section of the driver of the first vehicle, and the road condition of this section of road is very meaningful to the safe driving of the first vehicle, and it is a section that the driver of the first vehicle needs to pay attention to, therefore, obtaining a panoramic road condition image of this section of road plays an important role in improving the safe driving of the first vehicle. In addition, since the farthest visible position in front of the first vehicle is determined based on the real-time collected image of the front of the vehicle, it can change according to the weather factors or road congestion conditions based on the current driving of the first vehicle, and therefore, compared with fixed parameter values, it has greater flexibility.

[0039] For example, Figure 2 As shown, when the first vehicle is at position a, the farthest visible position in front of the first vehicle is b, the first vehicle is at a sharp turn, the distance from a to b is only 20 meters, and the end point of the preset distance range in front of the first vehicle is position c at 500 meters. Then, the above-mentioned panoramic road condition image with the farthest position as the starting point and within the preset distance range in front of the first vehicle is the panoramic road condition image of the road from b to c.

[0040] In an embodiment of the present application, an image in front of a vehicle captured by a first vehicle is obtained, and the farthest visible position in front of the first vehicle is determined based on the image in front of the vehicle. Then, multiple real-time images captured by a second vehicle with the farthest position as the starting point and located within a preset distance range in front of the first vehicle are obtained, and the multiple real-time images captured by the second vehicle are spliced ​​to obtain a panoramic road condition image with the farthest position as the starting point and located within the preset distance range in front of the first vehicle. The panoramic road condition image is displayed by the first vehicle, so that the driver of the first vehicle can predict the road condition in front of his vehicle in advance, which solves the problem that the driver is usually unable to promptly know the specific road condition in front of the vehicle during driving, which is conducive to solving traffic congestion, preventing traffic accidents and improving traffic safety.

[0041] In some embodiments of the present application, in the process of stitching the multiple real-time images collected by the second vehicle in the above step 103, the multiple real-time images collected by the second vehicle can be stitched using image stitching algorithms such as the Scale-Invariant Feature Transform (SIFT) algorithm, the Speeded Up Robust Features (SURF) algorithm, the feature matching algorithm, the As-Projective-As-Possible Image Stitching (APAP) algorithm, etc., and the present application does not impose any restrictions on this.

[0042] It should be noted that the first vehicle may also have the functions of the second vehicle, and the second vehicle may also have the functions of the first vehicle. That is, the first vehicle and the second vehicle may be vehicles with the same functions. After starting, the first vehicle may also collect image data of the vehicle's current environment and vehicle location information, and send the image data of the vehicle's current environment, vehicle location information, and image data acquisition timestamp to other vehicles or roadside equipment (for example, smart traffic lights, roadside cameras and other roadside equipment) through vehicle-to-vehicle interconnection technology (V2X), so that the roadside equipment can send them to other vehicles.

[0043] In an embodiment of the present application, when the above-mentioned road condition monitoring method is applied to a first vehicle, the above-mentioned road condition monitoring method may specifically refer to the first vehicle collecting an image in front of the vehicle through its own front camera configured in the background to determine the farthest visible position in front of it, and based on this, obtaining multiple real-time images collected by a second vehicle with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, so as to splice the multiple real-time images collected by the second vehicle, and then obtain a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, and the first vehicle displays the panoramic road condition image, so that the driver of the first vehicle can predict the road condition in front of his vehicle in advance, which solves the problem that the driver is usually unable to know the specific road conditions in front of the vehicle in time during driving, which is conducive to solving traffic congestion, preventing traffic accidents and improving traffic safety.

[0044] In some embodiments of the present application, after the first vehicle obtains a panoramic road condition image within a preset distance range in front of it, it can also send the panoramic road condition image to other vehicles adjacent to the first vehicle (for example, vehicles within 20 meters or 15 meters from the first vehicle), so that the drivers of the first vehicle and other vehicles adjacent to the first vehicle can predict the road conditions in front of their vehicles in advance, which solves the problem that drivers usually cannot promptly know the specific road conditions in front of the vehicle during driving, which is conducive to solving traffic congestion, preventing traffic accidents and improving traffic safety.

[0045] In an embodiment of the present application, when the above-mentioned road condition monitoring method is applied to a cloud server, the above-mentioned road condition monitoring method may specifically refer to: the cloud server receives an image of the front of the first vehicle captured by a front camera configured by itself to determine the farthest visible position in front of the first vehicle, and based on this, obtains multiple real-time images captured by a second vehicle with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, so as to splice the multiple real-time images captured by the second vehicle, and then obtain a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, and send the panoramic road condition image to the first vehicle, and the first vehicle displays the panoramic road condition image, so that the driver of the first vehicle can predict the road condition in front of his vehicle in advance, which solves the problem that the driver is usually unable to know the specific road condition in front of the vehicle in time during driving, which is conducive to solving traffic congestion, preventing traffic accidents and improving traffic safety.

[0046] Similarly, after the cloud server has spliced ​​together a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, it can also send the panoramic road condition image to other vehicles adjacent to the first vehicle (for example, vehicles within 20 meters or 15 meters from the first vehicle), so that the drivers of the first vehicle and other vehicles adjacent to the first vehicle can predict the road conditions in front of their vehicles in advance, solving the problem that drivers are usually unable to promptly know the specific road conditions in front of the vehicle during driving, which is conducive to solving traffic congestion, preventing traffic accidents and improving traffic safety.

[0047] As shown above, the panoramic road condition image within the preset distance range in front of the first vehicle can be spliced ​​by the first vehicle in its background as shown above, or it can be spliced ​​by the cloud server and then sent to the first vehicle. In order to prevent the driver of the first vehicle from being disturbed by the panoramic road condition image during driving, during the process of the first vehicle displaying the panoramic road condition image, the first vehicle can display the panoramic road condition image according to the front road condition image acquisition request triggered by the user (for example, the driver of the first vehicle) on the first vehicle. That is, the first vehicle displays the panoramic road condition image only after the user triggers the front road condition image acquisition request through voice control or by operating a physical button or a virtual button.

[0048] On this basis, in order to improve driving safety, in some embodiments of the present application, when the above-mentioned panoramic road condition image contains image content of a preset dangerous scene, the first vehicle can directly output danger warning information and display the panoramic road condition image without waiting for the user to trigger a request to obtain the road condition image ahead before displaying the panoramic road condition image.

[0049] For example, when the panoramic road condition image contains image content of preset dangerous scenes such as landslides, low visibility, or traffic accidents, severe congestion, etc., the first vehicle directly outputs danger warning information. For example, a warning sound indicating that there is a landslide on the road ahead is output, and the panoramic road condition image is displayed, so that the driver can know the road condition ahead in the first time and take countermeasures to reduce the occurrence of accidents.

[0050] In one embodiment of the present application, the judgment of whether the panoramic road condition image contains the image content of the preset dangerous scene can be determined by calculating the image similarity. For example, by comparing the panoramic road condition image with the sample images of various preset dangerous scenes collected in advance, when the similarity between the two is greater than the preset similarity threshold, it means that the panoramic road condition image contains the image content of the corresponding type of preset dangerous scene.

[0051] In one embodiment of the present application, for determining whether a panoramic road condition image contains image content of a preset dangerous scene, the panoramic road condition image can also be input into a second preset image recognition model, and the second preset image recognition model outputs a determination result of whether the panoramic road condition image contains image content of a preset dangerous scene, and when determining that the panoramic road condition image contains image content of a preset dangerous scene, outputs a recognition result of which type of preset dangerous scene it belongs to. The training process of the second preset image recognition model can refer to the training process of the first preset image recognition model, and will not be repeated here.

[0052] In one embodiment of the present application, in order to protect the privacy of the public, before stitching the multiple real-time images collected by the second vehicle in the above step 103, each of the multiple real-time images collected by the second vehicle may be desensitized.

[0053] In an embodiment of the present application, the above-mentioned desensitization processing may include but is not limited to mosaic processing of private content such as faces and license plate numbers contained in each real-time image.

[0054] In order to reduce the data volume of the panoramic road condition image, in one embodiment of the present application, in the above step 103, stitching the multiple real-time images collected by the second vehicle may include: filtering the multiple real-time images collected by the second vehicle based on a preset data correlation filtering strategy, and stitching the multiple real-time images collected by the second vehicle obtained by filtering.

[0055] In one embodiment of the present application, the filtering of the multiple real-time images collected by the second vehicle based on the preset data correlation filtering strategy includes: deleting the real-time images containing road conditions outside the preset distance range in front of the first vehicle among the multiple real-time images.

[0056] Specifically, based on the position of the second vehicle, the real-time images including the road condition content outside the preset distance range in front of the first vehicle among the multiple real-time images may be deleted.

[0057] For example, Figure 3As shown, assuming that position L is a position at a preset distance in front of the first vehicle, although the vehicle at position e is the second vehicle located within the preset distance range in front of the first vehicle, the image captured by the front camera of the vehicle at position e is a road condition image outside the preset distance range in front of the first vehicle (the distance between the vehicle at position e and position L is less than the set distance), so it can be deleted, but the content of the image captured by the rear camera of the vehicle at position e is a road condition image within the preset distance range in front of the first vehicle, so it needs to be retained. It should be noted that since the content of the image captured by the front camera of the vehicle at position d is the content contained in the field of view angle α, it already includes the road condition at position L, therefore, deleting the image captured by the front camera of the vehicle at position e will not affect the integrity of the panoramic road condition image finally stitched together.

[0058] In one embodiment of the present application, filtering the multiple real-time images collected by the second vehicle based on a preset data correlation filtering strategy includes: deleting the real-time images of normal road conditions from the multiple real-time images.

[0059] In the embodiment of the present application, normal road conditions refer to road conditions that do not affect normal driving. For example, real-time images that do not contain image content of preset dangerous scenes such as landslides, low visibility, or traffic accidents, severe congestion, and real-time images that do not contain image content such as pedestrians, obstacles, and traffic lights are real-time images corresponding to normal road conditions.

[0060] In the embodiment of the present application, since normal road conditions refer to road conditions that do not affect normal driving and are road conditions that the driver does not need to pay special attention to, they can be deleted to reduce the data volume of the panoramic road condition image.

[0061] In one embodiment of the present application, in order to make the display of the panoramic road condition image more targeted, during the process of the first vehicle displaying the panoramic road condition image, the image can also be displayed according to the user's setting parameters.

[0062] For example, when the driver only wants to know the image of the right lane, only the image of the right lane of the first vehicle may be displayed; when the driver only wants to know the image of the left lane, only the image of the left lane of the first vehicle may be displayed.

[0063] In order to improve the quality of the panoramic road condition image, in one embodiment of the present application, in the above step 103, stitching the multiple real-time images collected by the second vehicle may include: filtering the multiple real-time images collected by the second vehicle based on a preset image quality filtering strategy, and stitching the filtered multiple real-time images collected by the second vehicle.

[0064] For example, an image processing algorithm may be used to evaluate quality indicators such as clarity, brightness, and contrast of the real-time image collected by the second vehicle, and low-quality real-time images such as blurry, too dark, or too bright images may be deleted to obtain a filtered real-time image.

[0065] In one embodiment of the present application, before filtering the multiple real-time images collected by the second vehicle based on the preset image quality filtering strategy, the multiple real-time images collected by the second vehicle may also be subjected to image optimization processing, for example, image processing operations such as removing noise points in the real-time images and increasing the resolution of the real-time images, so as to more easily filter and obtain real-time images that meet the requirements.

[0066] In the embodiment of the present application, the above-mentioned road condition monitoring method can not only automatically detect the farthest visible position in front of the first vehicle, and obtain a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, such as Figure 4 As shown, the road condition monitoring method can also obtain a panoramic road condition image corresponding to the target location by executing steps 401 to 403 in response to a road condition image acquisition request for the target location triggered by the user.

[0067] Step 401: receiving a target location road condition image acquisition request triggered by a user on a first vehicle.

[0068] Step 402: acquiring a plurality of real-time images sent by a third vehicle located at the target location according to a target location road condition image acquisition request.

[0069] Step 403: stitching the multiple real-time images sent by the third vehicle to obtain a panoramic road condition image corresponding to the target position.

[0070] For example, when the driver of the first vehicle (vehicle A) wants to view the panoramic road image 1 km ahead (assuming it is point f), he can trigger a request to obtain the road image 1 km ahead through voice control or by operating a physical button or a virtual button. Assume that at this time, there are 4 vehicles (vehicles B, C, D and E) at point f taking pictures through cameras and sending the pictures they took through V2X technology. After receiving the pictures sent by these 4 vehicles, vehicle A processes and splices the pictures and finally displays them to the driver.

[0071] Specifically, the front cameras of vehicles B, C, D and E respectively capture images of point f, wherein vehicle B captures the front image of point a. Vehicle C captures the left image of point f. Vehicle D captures the right image of point f. Vehicle E captures the rear image of point f. Vehicles B, C, D and E send the captured image data and GPS location information to surrounding vehicles, including vehicle A, through V2V communication. After receiving the image data, vehicle A removes duplicate data, filters out old data older than 5 seconds, retains only the currently relevant images, and classifies the received image data, tags it with timestamps and location information. The onboard computer of vehicle A parses the image data according to the tag information, and aligns the data from different sources in time and space. Then, an image stitching algorithm is used to stitch the image data from vehicles B, C, D and E into a seamless panoramic road image, and the integrated panoramic road image is stored in the temporary storage of vehicle A, and the stitched panoramic road image is displayed in real time through the onboard display screen, and tag information is displayed, such as one or more tag information of data source, timestamp and location information. That is, vehicle A can receive pictures or video data of other surrounding vehicles through v2x technology, and perform time and space alignment based on the timestamp and corresponding positioning information, and use the image stitching algorithm to seamlessly stitch the aligned image data to generate a comprehensive panoramic road condition image.

[0072] In the embodiment of the present application, the specific implementation of the above step 403 can refer to the description of the above step 103, which will not be repeated here.

[0073] In practical applications, the third vehicle may be a vehicle having the same functions as the first vehicle and the second vehicle. In addition, the first vehicle, the second vehicle and the third vehicle may be equipped with a front camera, a rear camera, a side camera, a GPS device, a V2X communication module and a display module. Among them, the front camera and the rear camera are used to capture real-time images or videos of the front, rear and side of the vehicle; the GPS device is used to obtain the real-time location information of the vehicle; and the V2X communication module is used to realize vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2I) communication. Specifically, the vehicle (the first vehicle, the second vehicle and / or the third vehicle, this application does not limit this) sends the collected image or video data together with the GPS positioning information and GPS time to the surrounding vehicles through the V2V communication module. If there are not enough vehicles around, or in some key areas, the vehicle can transmit the data to the roadside infrastructure equipment, such as smart traffic lights, roadside cameras, etc., through the V2I communication module. Then the roadside infrastructure equipment sends it to other vehicles. The display module can be used to display the processed surrounding traffic conditions images or videos in real time.

[0074] In addition, to ensure the efficiency of data transmission, the vehicle can compress the collected and optimized images or videos. The compression algorithm can be selected from video compression algorithms such as H.264 or H.265. The vehicle can perform integrity checks on the received data to ensure that the data is not lost or damaged during transmission. If the data packet is found to be damaged or incomplete, request to resend the data. If multiple requests fail, abandon the request and discard the damaged data. The vehicle can also classify the received data according to the data source (vehicle or roadside infrastructure) and data type (image or video). Label each image or video data, including the data timestamp, location information (GPS coordinates), source vehicle ID or infrastructure ID, etc., so that each data packet contains a unique identifier (such as data packet ID, timestamp, source ID). When performing image stitching, the on-board computer can identify whether there are duplicate data packets by checking these identifiers. Or by calculating the hash value of each data packet (such as MD5 or SHA-256) and comparing the hash value of the received data packet to determine whether there is duplicate content. If the hash values ​​of two data packets are the same, they are considered to be duplicate data. For detected duplicate data packets, only one copy is retained and the rest of the duplicate data packets are discarded. Next, data that exceeds the preset time threshold (such as data from 10 seconds ago) is filtered out, and only the latest real-time data is retained. If the contents of multiple data packets are similar, the data packet with the latest timestamp is retained first. And based on the timestamp information, the image or video data from different vehicles and infrastructures are time-aligned to ensure the synchronization of the data in time. Using GPS positioning information, the data is spatially aligned to ensure the consistency of the data in geographical location. After the panoramic road condition image is stitched together, the stitched panoramic road condition image can be stored in the temporary memory of the on-board computer for subsequent processing and display.

[0075] It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, and in some embodiments of the present application, certain steps may be performed in other orders.

[0076] The present application also provides a road condition monitoring device, such as Figure 5 As shown, the road condition monitoring device 500 is configured in the first vehicle or the cloud server and may include:

[0077] A first acquisition unit 501 is used to acquire a vehicle front image collected by a first vehicle, and determine the farthest visible position in front of the first vehicle based on the vehicle front image;

[0078] The second acquisition unit 502 is used to acquire a plurality of real-time images collected by a second vehicle located within a preset distance range in front of the first vehicle with the farthest position as the starting point;

[0079] The stitching unit 503 is used to stitch multiple real-time images collected by the second vehicle to obtain a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, and the first vehicle displays the panoramic road condition image.

[0080] In one embodiment, the above-mentioned road condition monitoring device 500 may also include: a desensitization processing unit, which is used to perform desensitization processing on each of the multiple real-time images collected by the second vehicle; the desensitization processing includes mosaic processing of faces and license plate numbers contained in the real-time images.

[0081] In one embodiment, the stitching unit 503 is specifically used to filter the multiple real-time images collected by the second vehicle based on a preset data correlation filtering strategy and / or a preset image quality filtering strategy, and stitch the filtered multiple real-time images collected by the second vehicle.

[0082] In one embodiment, the above-mentioned filtering of multiple real-time images collected by the second vehicle based on a preset data correlation strategy includes: deleting real-time images containing road conditions outside a preset distance range in front of the first vehicle from the multiple real-time images; and / or deleting real-time images of ordinary road conditions from the multiple real-time images.

[0083] In one embodiment, the first vehicle displays the panoramic road condition image, including: the first vehicle displays the panoramic road condition image according to a received front road condition image acquisition request triggered by a user.

[0084] In one embodiment, the first vehicle displays the panoramic road condition image, including: when the panoramic road condition image contains image content of a preset dangerous scene, the first vehicle outputs danger warning information and displays the panoramic road condition image.

[0085] In one embodiment, the first acquisition unit 501 is also used to receive a target location road condition image acquisition request triggered by a user on the first vehicle; the second acquisition unit 502 is also used to acquire multiple real-time images sent by a third vehicle located at the target location according to the target location road condition image acquisition request; the stitching unit 503 is also used to stitch the multiple real-time images sent by the third vehicle to obtain a panoramic road condition image corresponding to the target location.

[0086] It should be noted that, for the convenience and simplicity of description, the specific working process of the road condition monitoring device 500 described above can refer to the above Figures 1 to 4 The corresponding process of the road condition monitoring method described in will not be repeated here.

[0087] like Figure 6 As shown, the embodiment of the present application further provides a terminal. The terminal may be a first vehicle, or a cloud server. Figure 6 As shown, the terminal 6 may include: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned various road condition monitoring method embodiments are implemented, for example, Figure 1 Steps 101 to 103 are shown.

[0088] The processor 60 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0089] The memory 61 may be an internal storage unit of the terminal 6, such as a hard disk or a memory. The memory 61 may also be an external storage device for the terminal 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal 6. Further, the memory 61 may also include both an internal storage unit of the terminal 6 and an external storage device. The memory 61 is used to store the above-mentioned computer program and other programs and data required by the terminal.

[0090] The computer program may be divided into one or more units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the process of the computer program executing the road condition monitoring method in the terminal.

[0091] For example, the above computer program can be divided into: a first acquisition unit, a second acquisition unit and a splicing unit, and the specific functions are as follows:

[0092] A first acquisition unit is used to acquire a vehicle front image collected by the first vehicle, and determine the farthest visible position in front of the first vehicle based on the vehicle front image;

[0093] A second acquisition unit is used to acquire a plurality of real-time images collected by a second vehicle located within a preset distance range in front of the first vehicle with the farthest position as a starting point;

[0094] The stitching unit is used to stitch multiple real-time images collected by the second vehicle to obtain a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, and the first vehicle displays the panoramic road condition image.

[0095] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0096] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0098] In the embodiments provided in the present application, it should be understood that the disclosed terminals and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic. For example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, systems or units, which can be electrical, mechanical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0101] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0102] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A road condition monitoring method, characterized in that: The road condition monitoring method is applied to a first vehicle or a cloud server, and the road condition monitoring method includes: Acquire a vehicle front image captured by the first vehicle, and determine the farthest visible position in front of the first vehicle based on the vehicle front image; Acquire multiple real-time images collected by a second vehicle located within a preset distance range in front of the first vehicle with the farthest position as a starting point; The multiple real-time images collected by the second vehicle are stitched together to obtain a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, and the first vehicle displays the panoramic road condition image.

2. The road condition monitoring method according to claim 1, characterized in that: Before stitching the multiple real-time images collected by the second vehicle, the road condition monitoring method includes: Desensitizing processing is performed on each of the multiple real-time images collected by the second vehicle; the desensitizing processing includes mosaic processing of faces and license plates contained in the real-time images.

3. The road condition monitoring method according to claim 1, characterized in that: The stitching of the multiple real-time images collected by the second vehicle comprises: The multiple real-time images collected by the second vehicle are filtered based on a preset data correlation filtering strategy and / or a preset image quality filtering strategy, and the multiple real-time images collected by the second vehicle obtained by filtering are spliced.

4. The road condition monitoring method according to claim 3, characterized in that: The filtering the multiple real-time images collected by the second vehicle based on a preset data correlation strategy includes: deleting the real-time images among the plurality of real-time images that include road conditions outside a preset distance range in front of the first vehicle; and / or, The real-time images of normal road conditions are deleted from the multiple real-time images.

5. The road condition monitoring method according to any one of claims 1 to 4, characterized in that: The first vehicle displays the panoramic road condition image, including: The first vehicle displays the panoramic road condition image according to a received front road condition image acquisition request triggered by a user.

6. The road condition monitoring method according to any one of claims 1 to 4, characterized in that: The first vehicle displays the panoramic road condition image, including: When the panoramic road condition image contains image content of a preset dangerous scene, the first vehicle outputs danger warning information and displays the panoramic road condition image.

7. The road condition monitoring method according to any one of claims 1 to 4, characterized in that: The road condition monitoring method further includes: Receiving a target location road condition image acquisition request triggered by a user on the first vehicle; Acquire multiple real-time images sent by a third vehicle located at the target location according to the target location road condition image acquisition request; The multiple real-time images sent by the third vehicle are stitched together to obtain a panoramic road condition image corresponding to the target position.

8. A road condition monitoring device, characterized in that: The road condition monitoring device is configured on the first vehicle or the cloud server, and the road condition monitoring device includes: A first acquisition unit, configured to acquire a vehicle front image collected by the first vehicle, and determine a farthest visible position in front of the first vehicle based on the vehicle front image; A second acquisition unit is used to acquire a plurality of real-time images collected by a second vehicle located within a preset distance range in front of the first vehicle with the farthest position as a starting point; A stitching unit is used to stitch multiple real-time images collected by the second vehicle to obtain a panoramic road condition image with the farthest position as the starting point and located within a preset distance range in front of the first vehicle, and the first vehicle displays the panoramic road condition image.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the road condition monitoring method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the road condition monitoring method according to any one of claims 1 to 7 are implemented.

Images