Road collapse identification method and device, intelligent equipment and computer program product

Abstract

The invention is suitable for the technical field of computers, and provides a road collapse identification method and device, intelligent equipment and a computer program product. The method comprises the following steps: acquiring a thermal image, a pixel image and point cloud data of a road in front of the intelligent equipment; based on the thermal image, determining a suspected collapse area on the front road; determining a target area in the pixel map and target point cloud data in the point cloud data, the target area being an area corresponding to the suspected collapse area in the pixel map, and the target point cloud data comprising a point cloud corresponding to the target area in the point cloud data; and performing collapse identification on the suspected collapse area based on the edge feature value and the color feature value of each pixel in the target area and the curvature feature value of each point cloud in the target point cloud data. According to the invention, the accuracy of road collapse identification can be improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and particularly relates to a method, device, intelligent device and computer program product for road collapse recognition. Background Art

[0002] Road collapse is a traffic hazard in a scenario, which not only causes economic losses, but also threatens the safety of intelligent devices (such as vehicles, mobile robots, etc.). Existing road collapse detection methods include manual inspection, fixed sensor monitoring, and drone inspection, etc. However, these methods have obvious limitations. On the one hand, the coverage range is limited, and on the other hand, the response speed is slow, and the accuracy of road collapse recognition is relatively low. Summary of the Invention

[0003] The embodiments of this application provide a method, device, intelligent device and computer program product for road collapse recognition, which can improve the accuracy of road collapse recognition.

[0004] In a first aspect, the embodiments of this application provide a method for road collapse recognition, including:

[0005] Obtain a thermal image, a pixel map and point cloud data of the road in front of the intelligent device;

[0006] Based on the thermal image, determine a suspected collapse area on the road in front;

[0007] Determine a target area in the pixel map and target point cloud data in the point cloud data, where the target area is the area corresponding to the suspected collapse area in the pixel map, and the target point cloud data includes the point cloud corresponding to the target area in the point cloud data;

[0008] Based on the edge feature value and color feature value of each pixel in the target area, and the curvature feature value of each point cloud in the target point cloud data, perform collapse recognition on the suspected collapse area.

[0009] In the embodiments of the present application, by acquiring the thermal imaging map, pixel map, and point cloud data of the road ahead of the intelligent device, it is possible to determine the suspected collapse area on the road ahead based on the thermal imaging map, and determine the target area in the pixel map and the target point cloud data in the point cloud data corresponding to the suspected collapse area. Based on the edge feature values and color feature values of each pixel in the target area, and the curvature feature values of each point cloud in the target point cloud data, a collapse identification is performed on the suspected collapse area. Since the thermal imaging map can reflect the temperature distribution state of the road surface, if there is a collapse on the road, it can be more obvious to find the phenomenon of discontinuous or fault temperature distribution from the thermal imaging map, and when there is a collapse on the road, the edge feature values, color feature values, and curvature feature values of the road surface will all change. Therefore, based on the thermal imaging map, it is possible to accurately identify the area where the temperature distribution on the road ahead is discontinuous or fault (i.e., the suspected collapse area). On this basis, combined with the edge feature values and color feature values of each pixel in the target area, and the curvature feature values of each point cloud in the target point cloud data, it is possible to further perform a collapse identification on the suspected collapse area, thereby improving the accuracy of road collapse identification.

[0010] In a second aspect, embodiments of the present application provide a road collapse identification device. This device can be applied to an intelligent device or the device itself serves as an intelligent device. The device includes:

[0011] A data acquisition module, configured to acquire the thermal imaging map, pixel map, and point cloud data of the road ahead of the intelligent device;

[0012] A region determination module, configured to determine the suspected collapse area on the road ahead based on the thermal imaging map;

[0013] A target determination module, configured to determine the target area in the pixel map and the target point cloud data in the point cloud data. The target area is the area corresponding to the suspected collapse area in the pixel map, and the target point cloud data includes the point cloud corresponding to the target area in the point cloud data;

[0014] A collapse identification module, configured to perform a collapse identification on the suspected collapse area based on the edge feature values and color feature values of each pixel in the target area, and the curvature feature values of each point cloud in the target point cloud.

[0015] In a third aspect, embodiments of the present application provide an intelligent device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the intelligent device implements the method according to any one of the above first aspects.

[0016] Fourthly, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a computer, the method described in the first aspect above is implemented.

[0017] Fifthly, an embodiment of the present application provides a computer program product including a computer program, and when the computer program is run, the method described in any one of the first aspect above is executed by an intelligent device.

[0018] It can be understood that the beneficial effects of the second to fifth aspects above can be referred to the relevant descriptions in the first aspect above, and will not be elaborated here. Description of the Drawings

[0019] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 is a schematic flowchart of the road collapse recognition method provided by an embodiment of the present application;

[0021] Figure 2 is an example diagram of counting the number of passing vehicles based on a roadside camera provided by an embodiment of the present application;

[0022] Figure 3-1 is an example diagram of a vehicle decelerating provided by an embodiment of the present application;

[0023] Figure 3-2 is another example diagram of a vehicle decelerating provided by an embodiment of the present application;

[0024] Figure 3-3 is yet another example diagram of a vehicle decelerating provided by an embodiment of the present application;

[0025] Figure 3-4 is still another example diagram of a vehicle decelerating provided by an embodiment of the present application;

[0026] Figure 4 is another schematic flowchart of the road collapse recognition method provided by an embodiment of the present application;

[0027] Figure 5 is a schematic structural diagram of the road collapse recognition device provided by an embodiment of the present application;

[0028] Figure 6 is a schematic structural diagram of the intelligent device provided by an embodiment of the present application. Detailed Embodiments

[0029] In the following description, specific details such as specific system architectures, technologies, etc. are presented for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0030] It should be understood that when used in the specification and appended claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0031] It should also be understood that the term "and / or" used in the specification and appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0032] As used in the specification and appended claims of the present application, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if detecting [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once detecting [the described condition or event]", or "in response to detecting [the described condition or event]" depending on the context.

[0033] In addition, in the description of the specification and appended claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0034] The reference to "one embodiment" or "some embodiments" etc. described in the specification of the present application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0035] Please refer to Figure 1 , Figure 1A flow chart of a road collapse identification method provided in an embodiment of the present application is shown. As an example and not limitation, the method is applied to an intelligent device (for example, the intelligent device may include a driving device, an intelligent car, a robot, etc.) or an electronic device. For example, the intelligent device is an intelligent car. The electronic device may be a server, a mobile phone, a tablet computer, a computer, or other related electronic device. For example, the server may also be a cloud server that communicates with the intelligent device. The intelligent car involved in the embodiment of the present application may include an unmanned car and an automatic driving car.

[0036] Specifically, Figure 1 The method provided in the embodiment of the present application is described by taking the method being applicable to a smart device as an example. Specifically, the method comprises the following steps:

[0037] Step 101, obtaining a thermal image, a pixel map, and point cloud data of the road ahead of the smart device.

[0038] Optionally, a thermal image may be obtained through an infrared thermal imaging camera installed on the smart device, a pixel image may be obtained through a photosensitive camera installed on the smart device, and point cloud data may be obtained through a laser radar installed on the smart device.

[0039] Step 102: Determine a suspected collapse area on the road ahead based on the thermal image.

[0040] Since thermal images can reflect the temperature distribution state of the road surface, if the road ahead collapses, it can be more clearly found from the thermal images that the temperature distribution is discontinuous or faulted. Therefore, based on the thermal images, the area with discontinuous or faulted temperature distribution on the road ahead can be accurately identified. Since thermal images do not have pixel information, it is difficult to identify the cause of the discontinuous or faulted temperature distribution in the area (it may be caused by road collapse or similar targets on the road). Therefore, the area with discontinuous or faulted temperature distribution identified based on thermal images can be called a suspected collapse area, which can be understood as an area where collapse is suspected or may occur.

[0041] In order to more accurately identify the cause of discontinuous temperature distribution or fault in the suspected collapse area, this embodiment can further perform collapse identification on the suspected collapse area based on pixel map and point cloud data, thereby solving the problem of low road collapse identification accuracy caused by using only a single camera.

[0042] Step 103, determining the target area in the pixel map and the target point cloud data in the point cloud data.

[0043] The target area is an area corresponding to the suspected collapsed area in the pixel map, and the target point cloud data includes a point cloud corresponding to the target area in the point cloud data.

[0044] By matching the thermal imaging map with the pixel map, the intelligent device can accurately determine, from the pixel map, the area that matches the area with discontinuous or tomographic temperature distribution in the thermal imaging map, and this area is the target area.

[0045] By matching the pixel map with the point cloud data, the intelligent device can accurately determine, from the point cloud data, the point cloud data that matches the target area, and this point cloud data is the target point cloud data.

[0046] Step 104: Based on the edge feature values and color feature values of each pixel in the target area, and the curvature feature values of each point cloud in the target point cloud data, perform collapse recognition on the suspected collapse area.

[0047] Among them, the above curvature feature values include but are not limited to principal curvature, Gaussian curvature, mean curvature, etc.

[0048] As an example rather than a limitation, the color feature values can be obtained by analyzing the color distribution of the target area in the pixel map. For example, for an RGB image, the color feature values of each pixel can be extracted from the RGB channels. The edge feature values can reflect the shape of the road collapse (for example, the road collapse is generally in a thin strip shape), and can be calculated by an edge detection algorithm. The curvature feature values of the point cloud can be estimated by performing eigen-decomposition on the covariance matrix of the local neighborhood point set of each point cloud.

[0049] The target point cloud data is the high-precision three-dimensional point cloud data of the front road surface provided by the lidar. By analyzing the curvature change of each point cloud in the target point cloud data, local depression or bulge features can be identified. If the road collapses, the road surface is usually uneven, and there is a height difference between different point clouds, which can be characterized by the curvature feature values. Therefore, collapse recognition can be performed through the curvature feature values of each point cloud. When the road collapses, the edge feature values and color feature values of the collapsed area are different from those of the non-collapsed area. Therefore, collapse recognition can be performed through the edge feature values and color feature values of the pixels.

[0050] Based on the edge feature values and color feature values of each pixel in the target area, and the curvature feature values of each point cloud in the target point cloud data, the intelligent device can perform data fusion on the target area and the target point cloud data, and further perform collapse recognition on the suspected collapse area, improving the accuracy of road collapse recognition.

[0051] In a possible implementation manner, the above step 102 may include:

[0052] Determine the first matching degree, the second matching degree, and the third matching degree. The first matching degree is the matching degree between the edge feature value of each pixel and the edge feature value representing road collapse. The second matching degree is the matching degree between the color feature value of each pixel and the color feature value representing road collapse. The third matching degree is the matching degree between the curvature feature value of each point cloud and the curvature feature value representing road collapse;

[0053] Based on the first matching degree, the second matching degree, and the third matching degree, perform collapse identification on the suspected collapse area.

[0054] Among them, the edge feature value representing road collapse can be the edge feature value of the pixels in the area where road collapse occurs. The color feature value representing road collapse can be the color feature value of the pixels in the area where road collapse occurs. The curvature feature value representing road collapse can be the curvature feature value of the point cloud in the area where road collapse occurs.

[0055] Optionally, the edge feature value representing road collapse, the color feature value representing road collapse, and the curvature feature value representing road collapse can be preset.

[0056] As an example rather than a limitation, multiple pixel maps with road collapse can be used for training to obtain the edge feature value representing road collapse and the color feature value representing road collapse. Multiple point cloud data with road collapse can be used for training to obtain the curvature feature value representing road collapse.

[0057] As an example rather than a limitation, the first matching degree, the second matching degree, and the third matching degree can be calculated by any matching algorithm such as Euclidean distance, cosine similarity, etc. The type of the matching algorithm is not limited in this application.

[0058] In a possible implementation manner, the above-mentioned performing collapse identification on the suspected collapse area based on the first matching degree, the second matching degree, and the third matching degree includes:

[0059] Based on the first target weight of the pixel map and the second target weight of the point cloud data, weight the first matching degree, the second matching degree, and the third matching degree to obtain the fusion matching degree of the corresponding points in the suspected collapse area;

[0060] If there are points in the suspected collapse area whose fusion matching degree is greater than or equal to the target collapse threshold, it is determined that the suspected collapse area has collapsed;

[0061] If there are no points in the suspected collapse area whose fusion matching degree is greater than or equal to the target collapse threshold, it is determined that the suspected collapse area has not collapsed.

[0062] Based on the first target weight of the pixel map and the second target weight of the point cloud data, the intelligent device weights the first matching degree, the second matching degree, and the third matching degree, enabling data fusion of the target area and the target point cloud data. On this basis, further collapse recognition is performed on the suspected collapse area to improve the accuracy of road collapse recognition.

[0063] The intelligent device can perform fusion matching through the following formula.

[0064] Fusion matching degree = WL * Third matching degree + WC * First matching degree + WC * Second matching degree

[0065] Where, WL represents the second target weight of the point cloud data; WC represents the first target weight of the pixel map.

[0066] Optionally, the first target weight of the pixel map, the second target weight of the point cloud data, and the target collapse threshold can be set in advance. By way of example and not limitation, the weight of the pixel map is 0.5, the weight of the point cloud data is 0.5, and the collapse threshold is 1.

[0067] Of course, it can be understood that the first target weight, the second target weight, and the target collapse threshold can also be determined in the following three ways.

[0068] Method 1: In the scenario where the intelligent device is driving on a highway or elevated road, if it detects that the target objects on the highway or elevated road are discontinuous or missing, the first target weight is obtained according to the first preset weight, the second target weight is obtained according to the second preset weight, and the target collapse threshold is obtained according to the first collapse threshold. The target objects are contour markers or guardrails. Among them, the first target weight is less than the first preset weight, the second target weight is greater than the second preset weight, and the first collapse threshold is greater than the target collapse threshold. If it detects that the target objects on the highway or elevated road are continuous and not missing, the first preset weight can be determined as the first target weight, and the second preset weight can be determined as the second target weight.

[0069] Based on the regulations for road construction, on highways or elevated roads, continuous guardrails and outline markers are required. In this embodiment, the continuity of the outline markers can be identified through a photosensitive camera, and the continuity of the guardrails on highways or elevated roads can be identified through a millimeter-wave radar. Since the road surfaces of highways or elevated roads are usually relatively flat, when it is identified that the outline markers or guardrails are discontinuous or missing, it indicates that there is an abnormality on the road surface. By obtaining the first target weight according to the first preset weight and the second target weight according to the second preset weight, the contribution degree of the pixel map in road collapse identification can be reduced, and the contribution degree of the point cloud data in road collapse identification can be increased. Thus, the false alarm rate caused by misidentification of the photosensitive camera can be reduced, and the accuracy of road collapse identification can be improved. By obtaining the target collapse threshold according to the first collapse threshold, the judgment threshold in road collapse identification can be reduced, thereby reducing the possibility of missed identification in the scenario where the road has an abnormality. As an example but not a limitation, both the first preset weight and the second preset weight are 0.5, the first collapse threshold is 1, the first target weight can be 0.4, the second target weight can be 0.6, and the target collapse threshold can be 0.8.

[0070] Method 2: Obtain the distance and average driving speed of the target section. The target section is the section between the first roadside image acquisition device and the second roadside image acquisition device. The first roadside image acquisition device and the second roadside image acquisition device are any two adjacent roadside image acquisition devices among two or more adjacent roadside image acquisition devices in the preset range of the front road.

[0071] Based on the distance and average driving speed of the target section, determine the passing time.

[0072] Based on the passing time and the first time period, determine the second time period. The start time of the second time period is later than the end time of the first time period. The duration between the start time of the first time period and the start time of the second time period is the passing time, and the duration between the end time of the first time period and the end time of the second time period is the passing time.

[0073] Determine the number of first passing intelligent devices and the number of second passing intelligent devices. The number of first passing intelligent devices is the number of passing intelligent devices counted by the first roadside image acquisition device in the first time period.

[0074] If the difference between the number of first passing intelligent devices and the number of second passing intelligent devices is greater than the difference threshold, then obtain the target collapse threshold according to the second collapse threshold, and the second collapse threshold is greater than the target collapse threshold; if the difference between the number of first passing intelligent devices and the number of second passing intelligent devices is less than or equal to the difference threshold, then the second collapse threshold can be determined as the target collapse threshold.

[0075] The intelligent device can combine the traffic congestion information of the target road section provided by the electronic map manufacturer. When there is no congestion on the target road section, it can determine the average driving speed of the passing intelligent device on the target road section. Dividing the distance between the target road sections by the average driving speed can obtain the passing duration.

[0076] As an example rather than a limitation, a certain moment after determining the passing duration can be determined as the start moment of the first time period. Based on the specified duration of the first time period, the end moment of the first time period can be determined. Based on the start moment of the first time period and the passing duration, the start moment of the second time period can be determined. Based on the end moment of the first time period and the passing duration, the end moment of the second time period can be determined.

[0077] Optionally, a difference threshold and a second collapse threshold can be set in advance.

[0078] In this embodiment, by obtaining the target collapse threshold according to the second collapse threshold, the judgment threshold in road collapse recognition can be reduced, thereby reducing the possibility of missed recognition in the scenario where the road has abnormalities.

[0079] The above road-side image acquisition device can be a road-side camera.

[0080] In an application scenario, the intelligent device can use the vehicle-side V2X technology and equipment to receive the image data captured by the road-side cameras 2-5 km ahead on the road, as well as the longitude, latitude and parameter information of the road-side cameras themselves (such as frame rate and resolution, etc.). Through the longitude, latitude and parameter information of the road-side cameras themselves, determine the actual sorting position of each road-side camera on the road ahead and the positional relationship with the intelligent device of the vehicle itself. The vehicle-side uses an image recognition algorithm to recognize the actual number of passing intelligent devices in each image collected by the road-side cameras, and compares the number of passing intelligent devices captured by two adjacent road-side cameras. At the same time, combined with the traffic congestion information of the map manufacturer, when the road traffic situation is smooth, it is found that the number of passing intelligent devices decreases significantly in each detection cycle. As Figure 2 shown is an example diagram of counting the number of passing intelligent devices based on road-side cameras provided by an embodiment of the present application. Figure 2 The vehicle itself in it is the intelligent device itself. Camera 1 counts the number of passing intelligent devices from T-10 to T. Based on the distance between the cameras and the average driving speed of the passing intelligent devices, such as the camera distance is 2000 m and the average driving speed of the passing intelligent devices is 120 kph, then Camera 2 counts the number of passing intelligent devices from T+50 to T+60. If the number of passing intelligent devices counted by Camera 2 is significantly less than the number of passing intelligent devices counted by Camera 1, it is considered that an abnormality has occurred in the road section between Camera 1 and Camera 2, and the second collapse threshold is dynamically adjusted based on this abnormality to obtain the target collapse threshold.

[0081] It should be understood that the first collapse threshold in Method 1 and the second collapse threshold in Method 2 may be the same or different, and the present application does not limit this. The target collapse threshold obtained in Method 1 and the target collapse threshold obtained in Method 2 may be the same or different, and the present application does not limit this.

[0082] Method 3: If the slope of the road ahead is greater than the slope threshold, the first target weight is obtained according to the third preset weight, and the second target weight is obtained according to the fourth preset weight. The first target weight is less than the third preset weight, and the second target weight is greater than the fourth preset weight. If the slope of the road ahead is less than or equal to the slope threshold, the third preset weight may be determined as the first target weight, and the fourth preset weight may be determined as the second target weight.

[0083] For the ramp scenario, since the slope of the road will cause differences in the recognition of road flatness, by combining the information of the high-precision map and the pitch angle of the vehicle itself, the slope of the road ahead can be determined. Because in a scenario with a large slope, due to the vehicle having a certain pitch angle, there will be a certain matching area in a normal scenario, but this matching area is usually regular, complete, and continuous. In this scenario, by obtaining the first target weight according to the third preset weight and the second target weight according to the fourth preset weight, the contribution degree of the pixel map in road collapse recognition can be reduced, and the contribution degree of the point cloud data in road collapse recognition can be increased, thereby reducing the false alarm rate caused by misrecognition of the photosensitive camera and improving the accuracy of road collapse recognition.

[0084] It should be understood that the first preset weight in Method 1 and the third preset weight in Method 3 may be the same or different, and the present application does not limit this. The second preset weight in Method 1 and the fourth preset weight in Method 3 may be the same or different, and the present application does not limit this. The first target weight obtained in Method 1 and the first target weight obtained in Method 3 may be the same or different, and the present application does not limit this. The second target weight obtained in Method 1 and the second target weight obtained in Method 3 may be the same or different, and the present application does not limit this.

[0085] As an example rather than a limitation, both the third preset weight and the fourth preset weight are 0.5, the first target weight may be 0.3, and the second target weight may be 0.7.

[0086] In a possible implementation manner, when it is determined that a suspected collapse area has collapsed, it further includes:

[0087] Obtain the distance between the intelligent device and the suspected collapse area;

[0088] If the distance between the smart device and the suspected collapse area is greater than or equal to the distance threshold, then based on the current driving speed of the smart device, determine the sum of the distance traveled by the smart device in the first time period and the distance traveled in the second time period to obtain a distance function, where the first time period and the second time period are unknown quantities, the first time period is the time period for deceleration at the first deceleration, the second time period is the time period for deceleration at the second deceleration, and the first deceleration is greater than the second deceleration;

[0089] Under the constraint that the value of the distance function is less than or equal to the distance between the smart device and the suspected collapse area, solving the first duration and the second duration in the distance function so that the first duration obtains a maximum value and the second duration obtains a minimum value;

[0090] Based on the maximum and minimum values, the smart device is controlled to decelerate.

[0091] The smart device can obtain the distance between the smart device and the suspected collapse area based on the three-dimensional coordinates of the target point cloud data.

[0092] Optionally, the distance threshold, the first deceleration and the second deceleration may be preset. The present application does not limit the specific values ​​of the distance threshold, the first deceleration and the second deceleration. For example, the distance threshold may be 120m or 90m, the first deceleration may be -1m / s 2 , the second deceleration is -8m / s 2 .

[0093] It should be understood that the second deceleration can also be adjusted based on the current driving speed of the smart device. When the current driving speed of the smart device is higher, the corresponding second deceleration is lower than when the current driving speed of the smart device is lower. As an example but not a limitation, if the current driving speed of the smart device is greater than or equal to the speed threshold (e.g., 100 kph), the second deceleration is determined to be -8 m / s. 2 If the current speed of the smart device is less than the speed threshold, the second deceleration is determined to be -5m / s 2 .

[0094] When a smart device detects a road abnormality or determines that a collapse has occurred in a suspected collapse area, it can send a reminder message to the driver to alert the driver that the road ahead has collapsed. It can also transmit information such as road abnormalities or road collapse to other surrounding smart devices through V2X technology to remind other smart devices to issue early warnings.

[0095] If the distance between the smart device and the suspected collapse area is less than the distance threshold, the smart device can be directly decelerated at the second deceleration rate to quickly reduce the driving speed of the smart device and avoid the smart device from entering the collapse area as much as possible.

[0096] like Figure 3-1The following is an exemplary diagram of the decelerated driving of the intelligent device provided by the embodiment of the present application. As Figure 3-2 The following is another exemplary diagram of the decelerated driving of the intelligent device provided by the embodiment of the present application. As Figure 3-3 The following is yet another exemplary diagram of the decelerated driving of the intelligent device provided by the embodiment of the present application. As Figure 3-4 The following is still another exemplary diagram of the decelerated driving of the intelligent device provided by the embodiment of the present application.

[0097] As Figure 4 The following is another schematic flowchart of the road collapse recognition method provided by the embodiment of the present application. As Figure 4As shown, an intelligent device (such as a vehicle) can obtain a thermal image of the road ahead through an infrared thermal imaging camera, and through a road flatness recognition algorithm, identify the precise area where the temperature distribution of the road part in the thermal image is discontinuous or has a fault (i.e., the suspected collapse area). Obtain a pixel map of the road ahead through a photosensitive camera, and through the matching algorithm of the thermal image and the pixel map, accurately match the area where the temperature distribution of the road part in the thermal image is discontinuous or has a fault on the pixel map. Obtain the point cloud data of the road ahead through a lidar, and through the matching algorithm of the pixel map and the point cloud data, match the point cloud data of the accurately matched area in the pixel map, and based on the result of the point cloud data, obtain the precise coordinates of the road collapse, and at the same time transmit the precise point cloud data (i.e., the target point cloud data) of the matching area (i.e., the target area) to the road collapse confirmation and specific type recognition algorithm module. Perform data fusion on the pixels in the accurately matched area of the pixel map and the target point cloud data of the accurate lidar in the matching area to confirm the road collapse and identify the type of the road collapse, and output the recognition result of the road collapse and the type of the road collapse. Based on the output road collapse recognition result, the type of the road collapse, and the precise coordinates of the road collapse (the distance between the vehicle and the suspected collapse area can be determined based on the precise coordinates of the road collapse), take corresponding response strategies for the vehicle (i.e., the strategy of controlling the vehicle to decelerate). In the high-speed road or elevated scenario, identify the continuity of the contour marks through a photosensitive camera, identify the continuity of the high-speed road / elevated guardrail through a millimeter-wave radar, and send the continuity recognition result to the road collapse confirmation and specific type recognition algorithm to adjust the weights and collapse thresholds in the algorithm. The vehicle can use a road anomaly recognition algorithm to perform anomaly recognition on the road ahead based on the images (i.e., pictures) collected by the roadside camera, the longitude and latitude of the roadside camera, and the parameter information, and send the road anomaly result and the location where the anomaly occurs to the road collapse confirmation and specific type recognition algorithm to adjust the collapse threshold in the algorithm. For the ramp scenario, since the slope of the road will cause differences in the recognition of the road collapse, the road ramp information of the high-precision map and the pitch angle information of the vehicle itself can be input into the ramp scenario recognition algorithm to obtain the ramp scenario information (i.e., the slope of the road), and the ramp scenario information is input into the road collapse confirmation and specific type recognition algorithm to adjust the weights in the algorithm, reduce the increase in the false alarm rate caused by the misrecognition of the camera, and improve the detection accuracy. Among them, the vehicle can measure its own pitch angle information through an Inertial Measurement Unit (IMU).

[0098] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution is prior or posterior, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0099] Corresponding to the road collapse recognition method described in the above embodiments, Figure 5 FIG. shows a schematic structural diagram of a road collapse recognition device provided by an embodiment of the present application. For the sake of convenience of description, only the parts related to the embodiments of the present application are shown.

[0100] Referring to Figure 5 , the device includes:

[0101] A data acquisition module 501, configured to acquire a thermal image, a pixel map, and point cloud data of the road ahead of the intelligent device;

[0102] A region determination module 502, configured to determine a suspected collapse region on the road ahead based on the thermal image;

[0103] A target determination module 503, configured to determine a target region in the pixel map and target point cloud data in the point cloud data, where the target region is the region corresponding to the suspected collapse region in the pixel map, and the target point cloud data includes the point cloud corresponding to the target region in the point cloud data;

[0104] A collapse recognition module 504, configured to perform collapse recognition on the suspected collapse region based on the edge feature value and color feature value of each pixel in the target region, and the curvature feature value of each point cloud in the target point cloud data.

[0105] Optionally, the above-mentioned collapse recognition module 504 includes:

[0106] A matching degree determination unit, configured to determine a first matching degree, a second matching degree, and a third matching degree, where the first matching degree is the matching degree between the edge feature value of each pixel and the edge feature value representing road collapse, the second matching degree is the matching degree between the color feature value of each pixel and the color feature value representing road collapse, and the third matching degree is the matching degree between the curvature feature value of each point cloud and the curvature feature value representing road collapse;

[0107] A collapse recognition unit, configured to perform collapse recognition on the suspected collapse region based on the first matching degree, the second matching degree, and the third matching degree.

[0108] Optionally, the above-mentioned collapse recognition unit is specifically configured to:

[0109] Based on the first target weight of the pixel map and the second target weight of the point cloud data, weight the first matching degree, the second matching degree, and the third matching degree to obtain the fusion matching degree of the corresponding points in the suspected collapse region;

[0110] If there are points with a fusion matching degree greater than or equal to the target collapse threshold in the suspected collapse area, it is determined that the suspected collapse area has collapsed;

[0111] If there are no points with a fusion matching degree greater than or equal to the target collapse threshold in the suspected collapse area, it is determined that the suspected collapse area has not collapsed.

[0112] Optionally, the above device further includes:

[0113] A parameter adjustment module, configured to, in a scenario where the intelligent device is driving on a highway or an elevated road, if it detects that the targets on the highway or the elevated road are discontinuous or missing, obtain the first target weight according to a first preset weight, obtain the second target weight according to a second preset weight, and obtain the target collapse threshold according to a first collapse threshold, where the targets are outline markers or guardrails, and the first target weight is less than the first preset weight, the second target weight is greater than the second preset weight, and the first collapse threshold is greater than the target collapse threshold.

[0114] Optionally, the above device further includes:

[0115] A data acquisition module, configured to acquire the distance and average driving speed of a target section, where the target section is the section between a first roadside image acquisition device and a second roadside image acquisition device, and the first roadside image acquisition device and the second roadside image acquisition device are any two adjacent roadside image acquisition devices among two or more adjacent roadside image acquisition devices within a preset range of the front road;

[0116] A duration determination module, configured to determine the passing duration based on the distance of the target section and the average driving speed;

[0117] A time period determination module, configured to determine a second time period based on the passing duration and a first time period, where the start time of the second time period is later than the end time of the first time period, the duration between the start time of the first time period and the start time of the second time period is the passing duration, and the duration between the end time of the first time period and the end time of the second time period is the passing duration;

[0118] A quantity determination module, configured to determine the number of first passing intelligent devices and the number of second passing intelligent devices, where the number of first passing intelligent devices is the number of passing intelligent devices counted by the first roadside image acquisition device during the first time period, and the number of second passing intelligent devices is the number of passing intelligent devices counted by the second roadside image acquisition device during the second time period;

[0119] A threshold adjustment module, configured to obtain the target collapse threshold according to a second collapse threshold if the difference between the number of the first access intelligent devices and the number of the second access intelligent devices is greater than a difference threshold, where the second collapse threshold is greater than the target collapse threshold.

[0120] Optionally, the above device further includes:

[0121] A weight adjustment module, configured to obtain the first target weight according to a third preset weight and obtain the second target weight according to a fourth preset weight if the slope of the front road is greater than a slope threshold, where the first target weight is less than the third preset weight and the second target weight is greater than the fourth preset weight.

[0122] Optionally, the above device further includes:

[0123] A distance acquisition module, configured to acquire the distance between the intelligent device and the suspected collapse area when it is determined that the suspected collapse area has collapsed;

[0124] A function determination module, configured to determine the sum of the distances traveled by the intelligent device in a first duration and in a second duration based on the current traveling speed of the intelligent device to obtain a distance function if the distance between the intelligent device and the suspected collapse area is greater than or equal to a distance threshold, where the first duration and the second duration are unknowns, the first duration is the duration of decelerating at a first deceleration, the second duration is the duration of decelerating at a second deceleration, and the first deceleration is greater than the second deceleration;

[0125] A duration solving module, configured to solve the first duration and the second duration in the distance function under the constraint that the value of the distance function is less than or equal to the distance between the intelligent device and the suspected collapse area, so that the first duration obtains a maximum value and the second duration obtains a minimum value;

[0126] A deceleration control module, configured to control the intelligent device to decelerate based on the maximum value and the minimum value.

[0127] It should be noted that for the information interaction, execution process, etc. between the above devices / units, since they are based on the same concept as the method embodiment of the present application, their specific functions and the technical effects brought, please refer to the method embodiment part for details, and will not be elaborated here.

[0128] Figure 6 This is a schematic structural diagram of the intelligent device provided by the embodiment of the present application. As Figure 6 shown, the intelligent device 6 of this embodiment includes: at least one processor 60 ( Figure 6only one is shown), a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60. When the processor 60 executes the computer program 62, the steps in any of the above method embodiments are implemented.

[0129] The intelligent device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art can understand that Figure 6 merely an example of the intelligent device 6, which does not constitute a limitation on the intelligent device 6. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0130] The so-called processor 60 may be a central processing unit (CPU), and the processor 60 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0131] In some embodiments, the memory 61 may be an internal storage unit of the intelligent device 6, such as the hard disk or memory of the intelligent device 6. In other embodiments, the memory 61 may also be an external storage device of the intelligent device 6, such as a plug-in hard disk equipped on the intelligent device 6, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 61 may also include both the internal storage unit and the external storage device of the intelligent device 6. The memory 61 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program, etc. The memory 61 may also be used to temporarily store data that has been output or will be output.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment and will not be repeated here.

[0133] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above embodiment methods of this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the device / smart device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0134] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0135] Those of ordinary skill in the art will realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0136] In the embodiments provided in this application, it should be understood that the disclosed devices / smart devices and methods can be implemented in other ways. For example, the device / smart device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0139] Any relevant user personal information that may be involved in the embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, for reasonable purposes based on business scenarios, and processing the personal information actively provided by users during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with user authorization.

[0140] The user's personal information processed by the applicant may vary depending on the specific product / service scenario. It shall be subject to the specific scenario of the user's use of the product / service and may involve the user's account information, device information, driving information, vehicle information or other relevant information. The applicant will treat the user's personal information and its processing with a high degree of diligence.

[0141] The applicant attaches great importance to the security of the user's personal information and has taken security protection measures that meet industry standards and are reasonable and feasible to protect the user's information and prevent the personal information from being accessed, publicly disclosed, used, modified, damaged or lost without authorization.

Claims

1. A road collapse identification method, characterized in that: include: Obtain thermal images, pixel images, and point cloud data of the road ahead from smart devices; Based on the thermal imaging image, determining a suspected collapse area on the road ahead; Determine a target area in the pixel map and target point cloud data in the point cloud data, wherein the target area is an area in the pixel map corresponding to the suspected collapsed area, and the target point cloud data includes a point cloud corresponding to the target area in the point cloud data; Based on the edge feature value and color feature value of each pixel in the target area and the curvature feature value of each point cloud in the target point cloud data, collapse identification is performed on the suspected collapse area.

2. The method according to claim 1, characterized in that The performing collapse identification on the suspected collapse area based on the edge feature value and color feature value of each pixel in the target area and the curvature feature value of each point cloud in the target point cloud data includes: Determine a first matching degree, a second matching degree, and a third matching degree, wherein the first matching degree is a matching degree between an edge feature value of each pixel and an edge feature value representing a road collapse, the second matching degree is a matching degree between a color feature value of each pixel and a color feature value representing a road collapse, and the third matching degree is a matching degree between a curvature feature value of each point cloud and a curvature feature value representing a road collapse; Based on the first matching degree, the second matching degree and the third matching degree, collapse identification is performed on the suspected collapse area.

3. The method according to claim 2, characterized in that The performing collapse identification on the suspected collapse area based on the first matching degree, the second matching degree and the third matching degree includes: Based on the first target weight of the pixel map and the second target weight of the point cloud data, weighting the first matching degree, the second matching degree and the third matching degree to obtain a fusion matching degree of corresponding points in the suspected collapse area; If there is a point in the suspected collapse area whose fusion matching degree is greater than or equal to the target collapse threshold, it is determined that the suspected collapse area has collapsed; If there is no point in the suspected collapse area whose fusion matching degree is greater than or equal to the target collapse threshold, it is determined that the suspected collapse area has not collapsed.

4. The method according to claim 3, characterized in that: In a scenario where the smart device is traveling on a highway or an elevated road, before weighting the first matching degree, the second matching degree, and the third matching degree based on the first target weight of the pixel map and the second target weight of the point cloud data, the method further includes: If it is detected that a target object on a highway or an elevated road is discontinuous or missing, the first target weight is obtained according to the first preset weight, the second target weight is obtained according to the second preset weight, and the target collapse threshold is obtained according to the first collapse threshold, and the target object is a contour mark or a guardrail, wherein the first target weight is less than the first preset weight, the second target weight is greater than the second preset weight, and the first collapse threshold is greater than the target collapse threshold.

5. The method according to claim 3, characterized in that: Before weighting the first matching degree, the second matching degree, and the third matching degree based on the first target weight of the pixel map and the second target weight of the point cloud data, the method further includes: Obtaining the spacing and average driving speed of a target road section, wherein the target road section is a road section between a first road-end image acquisition device and a second road-end image acquisition device, wherein the first road-end image acquisition device and the second road-end image acquisition device are any two adjacent road-end image acquisition devices among two or more adjacent road-end image acquisition devices within a preset range of the road ahead; Determining the travel time based on the distance between the target road sections and the average travel speed; Based on the travel time and the first time period, a second time period is determined, the start time of the second time period is later than the end time of the first time period, the time between the start time of the first time period and the start time of the second time period is the travel time, and the time between the end time of the first time period and the end time of the second time period is the travel time; Determine a first number of smart devices for passing traffic and a second number of smart devices for passing traffic, wherein the first number of smart devices for passing traffic is the number of smart devices for passing traffic counted by the first road-end image acquisition device in the first time period, and the second number of smart devices for passing traffic is the number of smart devices for passing traffic counted by the second road-end image acquisition device in the second time period; If the difference between the first number of passable smart devices and the second number of passable smart devices is greater than a difference threshold, the target collapse threshold is obtained according to a second collapse threshold, and the second collapse threshold is greater than the target collapse threshold.

6. The method according to claim 3, characterized in that Before weighting the first matching degree, the second matching degree, and the third matching degree based on the first target weight of the pixel map and the second target weight of the point cloud data, the method further includes: If the slope of the road ahead is greater than the slope threshold, the first target weight is obtained according to the third preset weight, and the second target weight is obtained according to the fourth preset weight, the first target weight is less than the third preset weight, and the second target weight is greater than the fourth preset weight.

7. The method according to any one of claims 1 to 6, characterized in that: When it is determined that the suspected collapse area has collapsed, the method further includes: Obtaining a distance between the smart device and the suspected collapse area; If the distance between the smart device and the suspected collapse area is greater than or equal to a distance threshold, then based on the current driving speed of the smart device, determine the sum of the distance traveled by the smart device in a first time period and the distance traveled in a second time period to obtain a distance function, where the first time period and the second time period are unknown quantities, the first time period is the time period for deceleration at a first deceleration, the second time period is the time period for deceleration at a second deceleration, and the first deceleration is greater than the second deceleration; Under the constraint that the value of the distance function is less than or equal to the distance between the smart device and the suspected collapse area, solving the first duration and the second duration in the distance function so that the first duration obtains a maximum value and the second duration obtains a minimum value; Based on the maximum value and the minimum value, the smart device is controlled to decelerate.

8. A road collapse identification device, characterized in that: include: A data acquisition module, used to acquire thermal imaging images, pixel images and point cloud data of the road ahead of the smart device; An area determination module, configured to determine a suspected collapse area on the road ahead based on the thermal image; A target determination module, used to determine a target area in the pixel map and target point cloud data in the point cloud data, wherein the target area is an area in the pixel map corresponding to the suspected collapsed area, and the target point cloud data includes a point cloud corresponding to the target area in the point cloud data; The collapse identification module is used to perform collapse identification on the suspected collapse area based on the edge feature value and color feature value of each pixel in the target area and the curvature feature value of each point cloud in the target point cloud data.

9. An intelligent device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the smart device implements the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed, causes the method according to any one of claims 1 to 7 to be performed.

Images