Cloud road surface obstacle early warning method based on state delay updating

Abstract

The invention relates to a cloud road obstacle early warning method based on state delay update, which is applied to the technical field of obstacle detection, and comprises the following steps: taking a video frame in video stream data, in which an obstacle is detected for the first time, as an initial frame, and storing the state information of the obstacle to a historical obstacle list; taking a frame next to the initial frame as a current frame, determining a current obstacle list from the current frame, and determining an optimal corresponding relation between the current obstacle list and a historical obstacle list; based on the optimal corresponding relation, determining obstacles which are successfully matched, currently unsuccessfully matched and historically unsuccessfully matched, updating effective counts, failure counts and state identifications, based on the updated state identifications, performing early warning reporting on obstacles belonging to a newly appearing target and a disappearing target, and updating a historical obstacle list; and taking the next frame of the current frame as the current frame, and returning to determine the current obstacle list from the current frame until the last video frame. The accuracy of obstacle early warning can be improved.

Description

Technical Field

[0001] This application relates to the field of obstacle detection technology, and in particular to cloud-based road obstacle warning based on state hysteresis updates. Background Technology

[0002] Road obstacle detection and early warning methods face challenges in practical applications. The shape and size of obstacles have significant uncertainties, and a "segmentation first, detection later" strategy is usually adopted. However, the reliability of the output results based on segmentation methods is often difficult to assess. In addition, in complex and dynamic road environments, detection systems based on single-frame image input are prone to missed detections and false detections, directly affecting the reliability of obstacle warnings and posing a potential threat to vehicle driving safety. Summary of the Invention

[0003] To address the aforementioned technical problems, this application provides a cloud-based road obstacle warning method based on state hysteresis updates, comprising: Acquire real-time video stream data, take the video frame in the video stream data where an obstacle is first detected as the initial frame, and store the status information of the detected obstacle in the historical obstacle list; the status information includes valid count, invalid count and status identifier, the status identifier is used to identify whether the obstacle belongs to a newly appeared target, a historical target or a disappeared target; The next frame after the initial frame is taken as the current frame. The current obstacle list is determined from the current frame, and the optimal correspondence between the current obstacle list and the historical obstacle list is determined by a matching algorithm. Based on the optimal correspondence, obstacles that have been successfully matched, currently unmatched, and historically unmatched are identified, and the valid counts and failure counts corresponding to the obstacles are updated. The status identifiers of the obstacles are updated based on the valid counts, invalid counts, and status identifiers of the obstacles, and the status information of the obstacles in the historical obstacle list is also updated. Based on the updated status identifiers, warnings are issued and reported for obstacles that belong to newly appearing targets and disappeared targets; Take the next frame after the current frame as the current frame and return to the step of determining the current obstacle list from the current frame, until the current frame is the last video frame.

[0004] Optionally, updating the valid count and invalid count corresponding to the obstacle includes: For each successfully matched obstacle, the valid count corresponding to the obstacle is incremented by 1, and the invalid count corresponding to the obstacle is set to 0; wherein, the initial value of the valid count corresponding to the obstacle is 1; For obstacles that are not currently matched, the valid count corresponding to the obstacle is set to 1, and the invalid count corresponding to the obstacle is set to 0. For obstacles that failed to match in the past, increment the failure count corresponding to the obstacle by 1. The initial value of the failure count corresponding to the obstacle is 0. The update of the status identifier corresponding to the obstacle based on the valid count, invalid count, and status identifier includes: If the status identifier is empty and the valid count corresponding to the obstacle is greater than the first preset threshold, the status identifier corresponding to the obstacle is set as a newly appearing target; If the status identifier is a newly appeared target or a historical target, and the failure count corresponding to the obstacle is greater than the second preset threshold, the status identifier corresponding to the obstacle is set as a disappeared target; If the status identifier is a newly appeared target or a historical target, and the failure count corresponding to the obstacle is less than or equal to the second preset threshold, the status identifier corresponding to the obstacle is set as a historical target.

[0005] Optionally, updating the status information of obstacles in the historical obstacle list includes: If the updated obstacle's status identifier is a historical target, update the status information of the obstacles in the historical obstacle list; If the updated obstacle's status is identified as a newly appearing target, the obstacle's status information is added to the historical obstacle list; If the updated obstacle's status is marked as a disappeared target, the obstacle's status information is removed from the historical obstacle list.

[0006] Optionally, the state information further includes obstacle confidence; the method for determining obstacle confidence includes: During obstacle detection, a segmentation mask corresponding to each obstacle is obtained from the video frame, and the type of the segmentation mask is determined based on the shape of the segmentation mask; If the segmentation mask is of the single convex hull type, the target convex hull region is determined from the segmentation mask, and the maximum inscribed ellipse of the target convex hull region is determined; the boundary line of the target convex hull region is located inside the segmentation mask; The obstacle confidence is determined based on the confidence of all pixels in the maximum inscribed ellipse; If the segmentation mask is of the single-dimple type, the segmentation mask is skeletonized to obtain the mask skeleton, and the obstacle confidence is determined based on the confidence of all pixels located on the mask skeleton. If the type of the segmentation mask is a combination type containing multiple convex hull types, a combination type containing multiple concave types, or a combination type containing at least one convex hull type and at least one concave type, after determining the obstacle confidence level corresponding to each type in the combination type, the final obstacle confidence level is determined based on the obstacle confidence levels corresponding to multiple types.

[0007] Optionally, determining the type of the segmentation mask based on its shape includes: The number of connected components in the segmentation mask is determined to be a first number; The segmentation mask is processed using morphological opening operations, and the number of connected components in the processed segmentation mask is determined as the second number. If the first quantity and the second quantity are not equal, the type of the segmentation mask is determined to be a combination type containing multiple convex hull types, a combination type containing multiple concave types, or a combination type containing at least one convex hull type and at least one concave type. If the first quantity and the second quantity are equal, calculate the ratio of the pixel area of ​​the segmentation mask to the area of ​​the minimum convex hull that encloses the segmentation mask; If the ratio is greater than or equal to the preset ratio, the type of the segmentation mask is determined to be a single convex hull type; If the ratio is less than the preset ratio, the type of the segmentation mask is determined to be a single recess type.

[0008] Optionally, determining the target convex hull region from the segmentation mask includes: The segmentation mask is eroded using structural elements to obtain an eroded mask. Obtain the boundary line of the minimum convex hull region of the erosion mask; If the boundary line is not completely inside the segmentation mask, after reducing the structuring element, return to the step of using the structuring element to erode the segmentation mask; If the boundary line is completely inside the segmentation mask, the minimum convex hull region of the erosion mask is taken as the target convex hull region.

[0009] The technical solution provided in this application has the following advantages compared with the prior art: After acquiring real-time video stream data, obstacles are detected. Upon first detection, the obstacle is stored in a historical obstacle list. Subsequent detected obstacles in the current obstacle list are matched against the historical list to determine successful matches, currently unmatched obstacles, and historically unmatched obstacles. This updates the valid and invalid counts in the obstacle status information. The status identifier is updated based on the current valid and invalid counts and the status identifier. Based on the updated status identifier, warnings are reported for obstacles belonging to newly appearing or disappearing targets, and the status information of obstacles in the historical obstacle list is updated. This cloud-based road obstacle warning method, based on delayed status updates, comprehensively considers the obstacle status information in the current video frame and multiple previous video frames, avoiding missed and false detections, improving the accuracy of obstacle warnings, and ultimately enhancing vehicle driving safety. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a cloud-based road obstacle warning method based on state hysteresis update in an embodiment of this application; Figure 2 This is a schematic diagram of the intersection and union of two matching targets in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the delayed update of obstacle state in an embodiment of this application; Figure 4A This is a schematic diagram of the real-scene perception results in the embodiments of this application; Figure 4B for Figure 4A The corresponding confidence distribution of correct / incorrect classification pixels; Figure 4C for Figure 4A The confidence distribution at different distances from the center; Figure 5 This is a flowchart of an obstacle confidence determination method in an embodiment of this application; Figure 6A This is a schematic diagram of a convex hull type in an embodiment of this application; Figure 6BThis is a schematic diagram of one type of recess in an embodiment of this application; Figure 6C This is a schematic diagram of a combination type in an embodiment of this application; Figure 7 This is a schematic diagram of an obstacle confidence determination method in an embodiment of this application; Figure 8 This is a flowchart illustrating the process of determining the target convex hull region from the segmentation mask in an embodiment of this application. Figure 9A This is a schematic diagram illustrating the determination of the maximum inscribed ellipse corresponding to the segmentation mask of the convex hull type in an embodiment of this application. Figure 9B This is a schematic diagram of the skeletonization process for the segmentation mask of the recess type in the embodiments of this application; Figure 10 This is a schematic diagram of a cloud-based road obstacle warning device based on state hysteresis update in an embodiment of this application. Figure 11 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. Detailed Implementation

[0013] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0014] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0015] See Figure 1 , Figure 1 This is a flowchart of a cloud-based road obstacle warning method based on state hysteresis update in this application embodiment, including the following steps: Step S102: Obtain real-time video stream data, take the video frame in the video stream data where the obstacle is first detected as the initial frame, and store the status information of the detected obstacle in the historical obstacle list.

[0016] Starting with the first video frame in the video stream, obstacle detection is performed on that frame. If no obstacle is detected, processing continues with the next video frame until an obstacle is detected. The video frame in which an obstacle is first detected is called the initial frame. The status information of the obstacles detected in the initial frame is stored in the historical obstacle list. The obstacle status information may include: a unique identifier, a valid count, a failed count, location information, and a status flag. The status flag is used to identify whether the obstacle is a newly appeared target, a historical target, or a disappeared target. Since the obstacle appears for the first time in the initial frame, the valid count is 1, the failed count is 0, and the status flag can be set to empty.

[0017] Step S104: Take the next frame after the initial frame as the current frame.

[0018] The same processing is performed on each video frame after the initial frame, and the video frame being processed is the current frame. Here, we take the next frame after the initial frame as an example for explanation.

[0019] Step S106: Determine the current obstacle list from the current frame, and determine the optimal correspondence between the current obstacle list and the historical obstacle list through a matching algorithm.

[0020] If an obstacle exists in the initial frame, it will usually also exist in the next frame. Similarly, obstacle detection is performed on the next frame to obtain the current obstacle list. Assuming the current obstacle list contains n targets, then the bounding boxes of the obstacles... The historical obstacle list contains m targets, and the bounding boxes of the obstacles are... (m,n≥0).

[0021] Construct the spatial location consistency matching cost matrix: ; l ij The spatial location matching cost between the i-th target in the current obstacle list and the j-th target in the historical obstacle list can be calculated using the following formula: ; A1 is the area of ​​the intersection rectangle of the two matched targets, A2 is the area of ​​the union rectangle of the two matched targets, D1 is the distance between the center points of the two matched targets, and D2 is the diagonal distance of the union rectangle of the two matched targets. (See also...) Figure 2 .

[0022] By using the Hungarian matching algorithm to process the matching cost matrix M, the optimal correspondence between the current obstacle list and the historical obstacle list can be obtained.

[0023] Step S108: Based on the optimal correspondence, determine the obstacles that have been successfully matched, are currently not successfully matched, and have been unsuccessfully matched in the past, and update the valid count and failure count corresponding to the obstacles.

[0024] Based on the optimal correspondence, obstacles can be divided into three categories: successfully matched obstacles, currently unmatched obstacles, and historically unmatched obstacles. Currently unmatched obstacles are those that do not exist in the historical obstacle list but exist in the current obstacle list. Successfully matched obstacles are those that exist in both the historical and current obstacle lists. Historically unmatched obstacles are those that exist in the historical obstacle list but do not exist in the current obstacle list.

[0025] See Figure 3 , Figure 3 This is a schematic diagram illustrating the delayed update of obstacle status in an embodiment of this application. For a successfully matched obstacle, indicating that the obstacle is a persistent historical target, the valid count for the obstacle is incremented by 1, and the invalid count for the obstacle is set to 0, as shown below: ; in, Indicates a valid count. Indicates the failure count. Right side This refers to the valid count of the obstacle in the historical obstacle list. Increasing this valid count by 1 gives the current valid count of the obstacle. Left side As described in step S102, the initial value of the valid count corresponding to the obstacle is 1.

[0026] For obstacles that fail to match, indicating that the obstacle may be a newly appeared target or a false detection target, the valid count for the obstacle is set to 1, and the invalid count for the obstacle is set to 0, as shown below: ; For obstacles that failed to match in the past, it indicates that the obstacle may be a missing target or a missed target. The failure count for the obstacle will be incremented by 1. The initial value of the failure count for the obstacle is 0, as shown below: .

[0027] Right side This refers to the failure count of the obstacle in the historical obstacle list. Increasing this failure count by 1 gives the current failure count for the obstacle. Left side .

[0028] Step S110: Update the status identifier of the obstacle based on the valid count, invalid count and status identifier of the obstacle, and update the status information of the obstacle in the historical obstacle list.

[0029] Status identifiers refer to the status identifiers corresponding to obstacles in the historical obstacle list. These status identifiers can be updated based on the valid count, invalid count, and status identifier of the obstacle. For example, status identifiers can include: 0, 1, and 2, where 0 indicates the obstacle is a newly appearing target, 1 indicates the obstacle is a persistent historical target, and 2 indicates the obstacle is a disappeared target.

[0030] like Figure 3 As shown, the system first checks if the status flag is empty. If the status flag is empty, it further checks if the valid count corresponding to the obstacle is greater than a first preset threshold (e.g., 5). If it is greater than the first preset threshold, it means that the obstacle appears in multiple video frames, and the status flag corresponding to the obstacle is set as a newly appearing target. If it is less than or equal to the first preset threshold, no processing is performed.

[0031] If the status identifier is not empty, it indicates that a status identifier has been assigned to it in the previous loop, and the disappearing target identified in the previous loop has been removed. Therefore, the status identifier is either a newly appearing target or a historical target. Further, it is determined whether the failure count corresponding to the obstacle is greater than the second preset threshold. If it is greater than the second preset threshold, the status identifier corresponding to the obstacle is set to a disappearing target. If it is less than or equal to the second preset threshold, the status identifier corresponding to the obstacle is set to a historical target.

[0032] Optionally, if the updated obstacle's status identifier is a historical target, the obstacle's status information in the historical obstacle list is updated, that is, the obstacle's current valid count, invalid count, and status identifier are updated in the historical obstacle list. If the updated obstacle's status identifier is a newly appeared target, the obstacle's status information is added to the historical obstacle list; if the updated obstacle's status identifier is a disappeared target, the obstacle's status information is deleted from the historical obstacle list.

[0033] Step S112: Based on the updated status identifier, issue early warnings and report obstacles that belong to newly appearing targets and disappearing targets.

[0034] An obstacle belonging to a newly appearing target indicates that the obstacle has just appeared, and an obstacle belonging to a disappeared target indicates that the obstacle has disappeared. Since the state of these two types of obstacles has changed, an early warning report is issued for these two types of obstacles.

[0035] Step S114: Determine whether the current frame is the last video frame.

[0036] If the current frame is the last video frame, the process ends; if the current frame is not the last video frame, proceed to step S116.

[0037] Step S116: Take the next frame of the current frame as the current frame, and return to step S106.

[0038] By executing a loop, the status information of obstacles in the historical obstacle list is continuously updated, and early warnings are reported.

[0039] The state information of obstacles can also include obstacle confidence, used to evaluate the credibility of the output obstacle. In practical obstacle detection based on segmentation algorithms, the segmentation precision of the target boundary region is somewhat insufficient: within the boundary region, non-obstacle pixels are easily misclassified as obstacle pixels, thus interfering with the accurate output of the overall obstacle confidence. For example, Figure 4A The real-scene perception results shown are illustrated below, with the confidence distribution of correctly / incorrectly classified pixels as follows: Figure 4B As shown, the confidence distribution at different distances from the center is as follows: Figure 4C As shown in the figure, analysis of the confidence spatial distribution reveals that misclassified pixels and pixels off-center generally exhibit low confidence levels.

[0040] In this embodiment of the application, when calculating the overall confidence of an obstacle, the contribution of confidence in the boundary region and the internal structurally irregular region (such as the concave region) is suppressed as much as possible to improve the accuracy of obstacle confidence assessment.

[0041] See Figure 5 , Figure 5 This is a flowchart of an obstacle confidence determination method in an embodiment of this application, including the following steps: Step S502: During obstacle detection, the segmentation mask corresponding to each obstacle is obtained from the video frame, and the type of segmentation mask is determined based on the shape of the segmentation mask.

[0042] For segmentation masks of different objects, they can be divided into two basic types: convex hull type and concave type, as shown below. Figure 6A and Figure 6B As shown. For road obstacle segmentation and detection tasks, segmentation masks can be divided into single-class (i.e., masks containing only one convex hull type or one concave type) and combined-class (i.e., masks composed of multiple convex hull types, multiple concave types, or at least one convex hull type and at least one concave type). Figure 6A and Figure 6B The images shown are all single-class segmentation masks. Figure 6C This is a segmentation mask for composite classes.

[0043] Alternatively, the type of segmentation mask can be determined in the following ways: The number of connected components in the segmentation mask is determined as a first quantity. Morphological opening operations are applied to the segmentation mask, and the number of connected components in the processed segmentation mask is determined as a second quantity. If the first and second quantities are not equal, the type of the segmentation mask is determined to be a combination type containing multiple convex hull types, a combination type containing multiple concave types, or a combination type containing at least one convex hull type and at least one concave type.

[0044] If the first and second quantities are equal, the ratio of the pixel area of ​​the segmentation mask to the area of ​​the minimum convex hull that encloses the segmentation mask can be expressed as: ; This represents the pixel area of ​​the segmentation mask. This represents the minimum convex hull area of ​​the enclosing segmentation mask. If the ratio is greater than or equal to a preset ratio, the segmentation mask type is determined to be a single convex hull type; if the ratio is less than the preset ratio, the segmentation mask type is determined to be a single concave type.

[0045] Different types of segmentation masks determine obstacle confidence levels differently; see [link to relevant documentation]. Figure 7 , Figure 7 This is a schematic diagram of an obstacle confidence determination method in an embodiment of this application, which is determined based on two basic types of segmentation masks.

[0046] Step S504: If the segmentation mask is of the single convex hull type, determine the target convex hull region from the segmentation mask, and determine the maximum inscribed ellipse of the target convex hull region. Determine the obstacle confidence based on the confidence of all pixels in the maximum inscribed ellipse.

[0047] like Figure 6A As shown, the boundary line of the minimum convex hull region of the segmentation mask is usually not completely inside the segmentation mask. To avoid misclassifying non-obstacle pixels as obstacle pixels within the boundary region, the segmentation mask is processed through erosion, convex hull fitting, and other techniques to ensure that the boundary line of the target convex hull region is completely inside the segmentation mask. See also Figure 8 , Figure 8 The flowchart for determining the target convex hull region from the segmentation mask in this embodiment includes: Step S802: Use structuring elements to perform erosion processing on the segmentation mask to obtain an eroded mask.

[0048] First, the segmentation mask is eroded using a larger structuring element (e.g., 11×11) to eliminate its boundary recessed areas, thus obtaining the eroded mask.

[0049] Step S804: Obtain the boundary line of the minimum convex hull region of the erosion mask.

[0050] The Graham scan method is used to process the erosion mask to obtain the boundary line of its minimum convex hull region.

[0051] Step S806: Determine whether the boundary line is completely inside the segmentation mask.

[0052] If the boundary line is not completely inside the segmentation mask, proceed to step S808; if the boundary line is inside the segmentation mask, the convex hull has completed boundary pixel elimination, and proceed to step S810.

[0053] Step S808: After reducing the structural elements, return to step S802.

[0054] After reducing the structuring element, step S802 is re-executed to refine the erosion of the segmentation mask with a smaller structuring element (e.g., 3×3) and finally determine the target convex hull region.

[0055] Execute step S810, and take the minimum convex hull region of the erosion mask as the target convex hull region.

[0056] To ensure that the selected pixels are sufficiently uniform and representative, the maximum inscribed ellipse of the target convex hull region can be determined. For example, Figure 9A The determined maximum inscribed circle and the pixel confidence scores within the maximum inscribed ellipse range have higher accuracy, and therefore can be used to determine the confidence scores of obstacles. The confidence scores of all pixels within this maximum inscribed ellipse range are then obtained. Sort them and calculate median ,Will The output is the confidence level of the obstacle.

[0057] Step S506: If the segmentation mask is of the single-dimple type, the segmentation mask is skeletonized to obtain the mask skeleton, and the obstacle confidence is determined based on the confidence of all pixels located on the mask skeleton.

[0058] Since the mask skeleton is located at the center of the entire segmentation mask, to avoid misclassifying non-obstacle pixels as obstacle pixels within the boundary region, the confidence scores of all pixels located on the skeleton can be obtained. Sort them and calculate median ,Will The output is the confidence level of the obstacle. For example, Figure 9B The determined mask skeleton has higher confidence accuracy for pixels, and therefore can be used to determine the confidence of obstacles.

[0059] Step S508: If the type of the segmentation mask is a combination type containing multiple convex hull types, a combination type containing multiple concave types, or a combination type containing at least one convex hull type and at least one concave type, after determining the obstacle confidence corresponding to each type in the combination type, the final obstacle confidence is determined based on the obstacle confidence corresponding to multiple types.

[0060] For combined types, they can first be divided into single convex hull types and single concave types. The obstacle confidence scores for each convex hull type and each concave type are then determined using the methods in steps S504 and S506, respectively. A confidence score set is formed, and the average of the obstacle confidence scores for multiple types in the confidence score set is used as the final obstacle confidence score.

[0061] When calculating the overall confidence level of obstacles, different methods are used to suppress the contribution of confidence levels in the boundary region for convex hull and concave types, thereby improving the accuracy of obstacle confidence assessment.

[0062] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0063] This application also provides a cloud-based road obstacle early warning device based on state hysteresis update, see [link to relevant documentation]. Figure 10 The cloud-based road obstacle warning device 1000 based on state hysteresis update includes: The historical obstacle list determination module 1002 is used to acquire real-time video stream data, take the video frame in the video stream data where an obstacle is first detected as the initial frame, and store the status information of the detected obstacle in the historical obstacle list; the status information includes valid count, invalid count and status identifier; the status identifier is used to identify whether the obstacle belongs to a newly appeared target, a historical target or a disappeared target; The current frame determination module 1004 is used to take the next frame after the initial frame as the current frame; The optimal correspondence determination module 1006 is used to determine the current obstacle list from the current frame and determine the optimal correspondence between the current obstacle list and the historical obstacle list through a matching algorithm; The first update module 1008 is used to determine the obstacles that have been successfully matched, are currently not successfully matched, and have been unsuccessfully matched in the past, based on the optimal correspondence, and to update the valid count and failure count corresponding to the obstacles. The second update module 1010 is used to update the status identifier corresponding to the obstacle based on the valid count, failure count and status identifier corresponding to the obstacle, and to update the status information of the obstacle in the historical obstacle list. The early warning reporting module 1012 is used to report early warnings for obstacles that belong to newly appearing targets and disappearing targets based on the updated status identifiers; The current frame update module 1014 is used to take the next frame of the current frame as the current frame and return the optimal correspondence determination module 1006 until the current frame is the last video frame.

[0064] Optionally, the first update module 1008 is specifically used to determine, based on the optimal correspondence, obstacles that have been successfully matched, currently unmatched, and historically unmatched. For successfully matched obstacles, the valid count corresponding to the obstacle is incremented by 1, and the failure count corresponding to the obstacle is set to 0; wherein, the initial value of the valid count corresponding to the obstacle is 1; for currently unmatched obstacles, the valid count corresponding to the obstacle is set to 1, and the failure count corresponding to the obstacle is set to 0; for historically unmatched obstacles, the failure count corresponding to the obstacle is incremented by 1, and the initial value of the failure count corresponding to the obstacle is 0. The second update module 1010 is specifically used to set the status identifier of the obstacle as a newly appearing target if the status identifier is empty and the effective count corresponding to the obstacle is greater than the first preset threshold; if the status identifier is a newly appearing target or a historical target and the failure count corresponding to the obstacle is greater than the second preset threshold, set the status identifier of the obstacle as a disappeared target; if the status identifier is a newly appearing target or a historical target and the failure count corresponding to the obstacle is less than or equal to the second preset threshold, set the status identifier of the obstacle as a historical target.

[0065] Optionally, the second update module 1010 is specifically used to update the status information of obstacles in the historical obstacle list through the following steps: If the updated obstacle's status is marked as a historical target, update the obstacle's status information in the historical obstacle list; if the updated obstacle's status is marked as a newly appeared target, add the obstacle's status information to the historical obstacle list; if the updated obstacle's status is marked as a disappeared target, delete the obstacle's status information from the historical obstacle list.

[0066] Optionally, the status information also includes obstacle confidence; the cloud-based road obstacle warning device 1000 based on state hysteresis update also includes: an obstacle confidence determination module; The obstacle confidence determination module includes: The mask type determination submodule is used to segment the video frame to obtain the segmentation mask corresponding to each obstacle during the obstacle detection process, and determine the type of segmentation mask based on the shape of the segmentation mask; The first obstacle confidence determination submodule is used to determine the target convex hull region from the segmentation mask if the segmentation mask type is a single convex hull type, and to determine the maximum inscribed ellipse of the target convex hull region; the boundary line of the target convex hull region is completely inside the segmentation mask; and the obstacle confidence is determined based on the confidence of all pixels in the maximum inscribed ellipse. The second obstacle confidence determination submodule is used to skeletonize the segmentation mask if the segmentation mask type is a single concave type, obtain the mask skeleton, and determine the obstacle confidence based on the confidence of all pixels located on the mask skeleton. The third obstacle confidence determination submodule is used to determine the final obstacle confidence based on the obstacle confidence corresponding to each type in the combination type if the segmentation mask type is a combination type containing multiple convex hull types, a combination type containing multiple concave types, or a combination type containing at least one convex hull type and at least one concave type.

[0067] Optionally, the mask type determination submodule is used to segment the video frame to obtain the segmentation mask corresponding to each obstacle during obstacle detection, and determine the number of connected components of the segmentation mask as a first number; process the segmentation mask using morphological opening operation, and determine the number of connected components of the processed segmentation mask as a second number; if the first number and the second number are not equal, determine the type of the segmentation mask as a combination type containing multiple convex hull types, a combination type containing multiple concave types, or a combination type containing at least one convex hull type and at least one concave type; if the first number and the second number are equal, calculate the ratio of the pixel area of ​​the segmentation mask to the area of ​​the smallest convex hull that wraps the segmentation mask; if the ratio is greater than or equal to a preset ratio, determine the type of the segmentation mask as a single convex hull type; if the ratio is less than the preset ratio, determine the type of the segmentation mask as a single concave type.

[0068] Optionally, the first obstacle confidence determination submodule determines the target convex hull region from the segmentation mask through the following steps: The segmentation mask is eroded using structuring elements to obtain an eroded mask; Obtain the boundary line of the minimum convex hull region of the erosion mask; If the boundary line is not completely inside the segmentation mask, after reducing the structuring element, return to the step of using the structuring element to perform erosion processing on the segmentation mask; If the boundary line is located inside the segmentation mask, the smallest convex hull region of the erosion mask is taken as the target convex hull region.

[0069] The specific details of each module or unit in the above-mentioned device have been described in detail in the corresponding methods, so they will not be repeated here.

[0070] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0071] This application also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the cloud-based road obstacle warning method based on state hysteresis update described in this example embodiment.

[0072] Reference Figure 11 , Figure 11 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0073] like Figure 11 As shown, the electronic device may include: a processor 1102, a communication interface 1104, a memory 1106, and a communication bus 1108.

[0074] The processor 1102, communication interface 1104, and memory 1106 communicate with each other via communication bus 1108.

[0075] Communication interface 1104 is used to communicate with other electronic devices or servers.

[0076] The processor 1102 is used to execute program 1110, specifically the relevant steps in the above method embodiments.

[0077] Specifically, program 1110 may include program code that includes computer operation instructions.

[0078] The processor 1102 may be a central processing unit, a specific integrated circuit, or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0079] Memory 1106 is used to store program 1110. Memory 1106 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0080] Specifically, program 1110 can be used to cause processor 1102 to execute the steps in the above-described embodiment of the cloud-based road obstacle warning method based on state hysteresis update.

[0081] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0082] In this embodiment of the application, a computer-readable storage medium is also provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the above-described cloud-based road obstacle warning method based on state hysteresis update is implemented.

[0083] It should be noted that the computer-readable storage medium shown in this application can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0084] In this embodiment of the application, a computer program product is also provided, which, when running on a computer, causes the computer to execute the above-described cloud-based road obstacle warning method based on state hysteresis update.

[0085] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0086] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cloud-based road obstacle early warning method based on state hysteresis update, characterized in that, include: Acquire real-time video stream data, take the video frame in the video stream data where an obstacle is first detected as the initial frame, and store the status information of the detected obstacle in the historical obstacle list; the status information includes valid count, invalid count and status identifier, the status identifier is used to identify whether the obstacle belongs to a newly appeared target, a historical target or a disappeared target; The next frame after the initial frame is taken as the current frame. The current obstacle list is determined from the current frame, and the optimal correspondence between the current obstacle list and the historical obstacle list is determined by a matching algorithm. Based on the optimal correspondence, obstacles that have been successfully matched, currently unmatched, and historically unmatched are identified, and the valid counts and failure counts corresponding to the obstacles are updated. The status identifiers of the obstacles are updated based on the valid counts, invalid counts, and status identifiers of the obstacles, and the status information of the obstacles in the historical obstacle list is also updated. Based on the updated status identifiers, warnings are issued and reported for obstacles that belong to newly appearing targets and disappeared targets; Take the next frame after the current frame as the current frame and return to the step of determining the current obstacle list from the current frame, until the current frame is the last video frame.

2. The method according to claim 1, characterized in that, The update of the valid count and invalid count corresponding to the obstacle includes: For each successfully matched obstacle, the valid count corresponding to the obstacle is incremented by 1, and the invalid count corresponding to the obstacle is set to 0; wherein, the initial value of the valid count corresponding to the obstacle is 1; For obstacles that are not currently matched, the valid count corresponding to the obstacle is set to 1, and the invalid count corresponding to the obstacle is set to 0. For obstacles that failed to match in the past, increment the failure count corresponding to the obstacle by 1. The initial value of the failure count corresponding to the obstacle is 0. The update of the status identifier corresponding to the obstacle based on the valid count, invalid count, and status identifier includes: If the status identifier is empty and the valid count corresponding to the obstacle is greater than the first preset threshold, the status identifier corresponding to the obstacle is set as a newly appearing target; If the status identifier is a newly appeared target or a historical target, and the failure count corresponding to the obstacle is greater than the second preset threshold, the status identifier corresponding to the obstacle is set as a disappeared target; If the status identifier is a newly appeared target or a historical target, and the failure count corresponding to the obstacle is less than or equal to the second preset threshold, the status identifier corresponding to the obstacle is set as a historical target.

3. The method according to claim 1, characterized in that, Updating the status information of obstacles in the historical obstacle list includes: If the updated obstacle's status identifier is a historical target, update the status information of the obstacles in the historical obstacle list; If the updated obstacle's status is identified as a newly appearing target, the obstacle's status information is added to the historical obstacle list; If the updated obstacle's status is marked as a disappeared target, the obstacle's status information is removed from the historical obstacle list.

4. The method according to claim 1, characterized in that, The status information also includes obstacle confidence; Methods for determining obstacle confidence include: During obstacle detection, a segmentation mask corresponding to each obstacle is obtained from the video frame, and the type of the segmentation mask is determined based on the shape of the segmentation mask; If the segmentation mask is of the single convex hull type, the target convex hull region is determined from the segmentation mask, and the maximum inscribed ellipse of the target convex hull region is determined; the boundary line of the target convex hull region is located inside the segmentation mask; The obstacle confidence is determined based on the confidence of all pixels in the maximum inscribed ellipse; If the segmentation mask is of the single-dimple type, the segmentation mask is skeletonized to obtain the mask skeleton, and the obstacle confidence is determined based on the confidence of all pixels located on the mask skeleton. If the segmentation mask is a combination type containing multiple convex hull types, a combination type containing multiple concave types, or a combination type containing at least one convex hull type and at least one concave type, after determining the obstacle confidence level corresponding to each type in the combination type, the final obstacle confidence level is determined based on the obstacle confidence levels corresponding to multiple types.

5. The method according to claim 4, characterized in that, Determining the type of the segmentation mask based on its shape includes: The number of connected components in the segmentation mask is determined to be a first number; The segmentation mask is processed using morphological opening operations, and the number of connected components in the processed segmentation mask is determined as the second number. If the first quantity and the second quantity are not equal, the type of the segmentation mask is determined to be a combination type containing multiple convex hull types, a combination type containing multiple concave types, or a combination type containing at least one convex hull type and at least one concave type. If the first quantity and the second quantity are equal, calculate the ratio of the pixel area of ​​the segmentation mask to the area of ​​the minimum convex hull that encloses the segmentation mask; If the ratio is greater than or equal to the preset ratio, the type of the segmentation mask is determined to be a single convex hull type; If the ratio is less than the preset ratio, the type of the segmentation mask is determined to be a single recess type.

6. The method according to claim 4, characterized in that, Determining the target convex hull region from the segmentation mask includes: The segmentation mask is eroded using structural elements to obtain an eroded mask. Obtain the boundary line of the minimum convex hull region of the erosion mask; If the boundary line is not completely inside the segmentation mask, after reducing the structuring element, return to the step of using the structuring element to erode the segmentation mask; If the boundary line is completely inside the segmentation mask, the minimum convex hull region of the erosion mask is taken as the target convex hull region.

Images