A bridge edge construction risk identification method and device, electronic equipment and program product

Abstract

The present invention discloses a method and device for identifying risks of construction on the edge of a bridge, an electronic device and a program product. The method for identifying risks of construction on the edge of a bridge comprises: acquiring multiple position images; performing target object detection processing on the multiple position images to obtain detection frames and feature points of the target objects in the multiple position images; performing edge image segmentation processing on the multiple position images to obtain image segmentation results of the edge of the bridge construction side in the multiple position images; calculating and obtaining three-dimensional spatial information including at least the edge of the bridge construction side based on at least two position images in the multiple position images; mapping the feature points of the target object to the three-dimensional spatial information to obtain the three-dimensional coordinates of the feature points of the target object, and determining the shortest distance from the feature points of the target object to the edge of the bridge construction side; and generating a bridge edge construction risk identification signal when the shortest distance is less than or equal to a preset safety threshold.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to a method and device for identifying risks in bridge edge construction, electronic equipment and a program product. Background Art

[0002] During the process of bridge reconstruction, expansion or maintenance, it is often necessary to carry out high-altitude work near the edge of the bridge. Since the work area is adjacent to the area under the bridge with a large drop, if construction workers and construction equipment get too close to the edge without safety measures, it is very easy to cause safety accidents such as falling.

[0003] At present, the commonly used monitoring methods for bridge construction mostly rely on manual inspections, and lack sufficiently accurate spatial distance judgments for the spatial position of the construction side edge and the positions of construction personnel and equipment. Especially when personnel and equipment are often moving on the construction site and the environment is complex, it is difficult to timely and accurately identify and warn of potential bridge edge construction risks.

[0004] At present, there are also edge risk detection methods proposed. CN114373115A discloses an edge target detection method, including: collecting an image of the area to be detected; inputting the image into a feature extraction network as a backbone network to extract the depth features of the image; inputting the depth features of the image into a regional generation network module to obtain a target candidate frame; inputting the target candidate frame and the depth features of the image into a regional feature extraction module to obtain common features; transmitting the common features to a height difference prediction module to obtain an edge height feature map; transmitting the common features obtained in S14 to a first classification and regression module to obtain a classification feature map; fusing the edge height feature map with the classification feature map, and sending the fused feature map to a second classification and regression module to obtain a target detection frame. CN113833290A discloses a method and system for reducing safety hazards of safety protection of edge openings at construction sites. The system includes an opening protection module, a cover plate and a mobile monitoring device. The mobile monitoring device includes a status monitoring component and a controller. The status monitoring component is used to obtain status information of the cover plate. The controller is used to determine whether the cover plate is moved based on the status information. The controller stores first identification information of the cover plate. The controller is used to send early warning information and first identification information when it is determined that the cover plate is moved; a monitoring center module is used to receive the early warning information and the first identification information sent by the controller, determine whether the cover plate moves abnormally, and if so, send a warning message; a terminal module includes a mobile terminal, and the mobile terminal is used to receive the warning message.

[0005] However, these conventional edge risk detection methods are difficult to apply to bridge reconstruction, expansion or maintenance construction scenarios where the construction area is constantly changing and camera installation is restricted.

[0006] There is an urgent need for an effective edge risk identification solution suitable for bridge reconstruction and expansion construction scenarios to provide more effective technical support for bridge construction safety management.

[0007] The content of this background technology description is only for facilitating understanding of the relevant technology in this field and is not regarded as an admission of the prior art. Summary of the invention

[0008] The embodiments of the present invention aim to provide a solution that can at least partially solve the above-mentioned problems.

[0009] In a first aspect, an embodiment of the present invention provides a method for identifying risks in bridge edge construction, which may include:

[0010] Acquire a plurality of position images, wherein the plurality of position images are collected by a camera installed on the inspection robot when the inspection robot travels to different positions along the non-construction side guardrail of the bridge;

[0011] Performing target object detection processing on the multiple position images to obtain detection frames and feature points of the target objects in the multiple position images;

[0012] Performing edge image segmentation processing on the multiple position images to obtain image segmentation results of the edge of the bridge construction side in the multiple position images;

[0013] Reconstructing three-dimensional reconstruction space information including the construction side edge of the bridge based on at least two of the multiple position images;

[0014] Mapping the feature points of the target object to the three-dimensional reconstruction information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space, and determining the shortest distance from the feature points of the target object to the edge of the bridge construction side in the three-dimensional reconstruction space; and

[0015] When the shortest distance is less than or equal to a preset safety threshold, a bridge edge construction risk identification signal is generated.

[0016] In some embodiments, the method further comprises:

[0017] Acquire image frames at least two time intervals at each of the different positions;

[0018] Performing target object detection processing on the image frames at least two time intervals to obtain position information of the target object in the image frames at least two time intervals; and

[0019] The motion state of the target object is determined based on the position information.

[0020] In these embodiments, when the target object is in motion, mapping the feature points of the target object to the three-dimensional reconstruction information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space includes: performing motion displacement compensation on the three-dimensional position of the feature points of the target object based on the motion state of the target object.

[0021] In some embodiments, reconstructing the three-dimensional reconstruction information including the construction side edge of the bridge based on at least two position images among the multiple position images includes:

[0022] Extracting a scene reference point from each of the plurality of acquired location images, wherein the scene reference point includes a road marking feature point, a safety fence feature point, or a ground identification point;

[0023] Establishing the correspondence between scene reference points of multiple acquired position images to construct a three-dimensional space structure;

[0024] sequentially matching the scene reference points of the newly acquired position image with the three-dimensional space structure to update the three-dimensional space structure;

[0025] Mapping or updating the image segmentation result of the edge into the three-dimensional space structure to obtain three-dimensional reconstructed space information including the edge of the bridge construction side;

[0026] Mapping the feature points of the target object to the three-dimensional reconstruction space information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space includes:

[0027] Acquire pixel coordinates of a feature point of the target object in at least one position image of a plurality of position images;

[0028] Based on the spatial transformation relationship between pixel coordinates or based on the mapping relationship between pixel coordinates and depth, the three-dimensional position of the feature point of the target object in the three-dimensional reconstruction space is determined.

[0029] In some embodiments, when the target object is in motion, mapping the feature points of the target object to the three-dimensional reconstruction information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space includes:

[0030] Acquire pixel coordinates of a feature point of the target object in at least one position image of a plurality of position images;

[0031] Determine the motion speed of the target object based on the pixel difference of the feature point of the target object in the image frames of the at least two time intervals and the time interval, and perform motion displacement compensation on the pixel coordinates according to the motion speed;

[0032] Based on the spatial transformation relationship between the compensated pixel coordinates or based on the mapping relationship between the compensated pixel coordinates and the depth, the three-dimensional position of the feature point of the target object in the three-dimensional reconstruction space is determined.

[0033] In some embodiments, reconstructing the three-dimensional reconstruction information including the construction side edge of the bridge based on at least two position images among the multiple position images includes:

[0034] Selecting two position images from the plurality of position images to form a spatial stereoscopic image pair;

[0035] Performing stereo matching calculation on the spatial stereo image pair to obtain scene depth information;

[0036] Obtaining depth information of the edge of the bridge construction side according to the image segmentation result of the edge in the spatial stereo image pair and combining the scene depth information;

[0037] Mapping the feature points of the target object to the three-dimensional reconstruction information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space includes:

[0038] Obtain pixel coordinates of feature points of the target object in the spatial stereo image pair;

[0039] Based on the pixel coordinates and the scene depth information, a three-dimensional position of the feature point of the target object in a three-dimensional reconstruction space is determined.

[0040] In some embodiments, when the target object is in motion, mapping the feature points of the target object to the three-dimensional reconstruction information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space includes:

[0041] Obtain pixel coordinates of feature points of the target object in the spatial stereo image pair;

[0042] Determine the motion speed of the target object based on the pixel difference of the target object in the image frames of the at least two time intervals and the time interval, and perform motion displacement compensation on the pixel coordinates according to the motion speed;

[0043] Based on the compensated pixel coordinates and the scene depth information, the three-dimensional position of the feature point of the target object in the three-dimensional reconstruction space is determined.

[0044] In a second aspect, an embodiment of the present invention provides a method for identifying risks in bridge edge construction, which may include:

[0045] Acquire binocular image pairs at multiple positions, wherein the binocular image pairs at the multiple positions are synchronously acquired by a pair of cameras installed on the inspection robot when the inspection robot travels to different positions along the non-construction side guardrail of the bridge;

[0046] Performing target object detection processing on at least one image in the binocular image pair at each position to obtain a detection frame of the target object and its feature points;

[0047] Performing edge image segmentation processing on at least one image in the binocular image pair at each position to obtain an image segmentation result of the edge of the bridge construction side;

[0048] Performing stereo matching calculation on the binocular image pair at each position to obtain scene depth information corresponding to the position, wherein the scene depth information includes depth information of the edge of the bridge construction side;

[0049] Determine the depth information of the feature point of the target object according to the pixel coordinates of the feature point of the target object at the position and the scene depth information, and determine the shortest distance from the feature point of the target object to the edge of the bridge construction side according to the depth information of the bridge construction side edge and the depth information of the feature point; and

[0050] When the shortest distance is less than or equal to a preset safety threshold, a bridge edge construction risk identification signal is generated.

[0051] In a third aspect, an embodiment of the present invention provides a device for identifying risks in bridge edge construction, comprising:

[0052] An image acquisition unit is configured to acquire a plurality of position images, wherein the plurality of position images are acquired by a camera installed on the inspection robot when the inspection robot travels to different positions along the non-construction side guardrail of the bridge;

[0053] a detection unit configured to perform target object detection processing on the multiple position images to obtain detection frames and feature points of the target objects in the multiple position images;

[0054] a segmentation unit configured to perform edge image segmentation processing on the plurality of position images to obtain image segmentation results of the edge of the bridge construction side in the plurality of position images;

[0055] A three-dimensional reconstruction unit, configured to reconstruct three-dimensional reconstruction space information including the edge of the bridge construction side based on at least two position images of the plurality of position images;

[0056] a mapping unit configured to map the feature point of the target object to the three-dimensional reconstruction information to obtain a three-dimensional position of the feature point of the target object in the three-dimensional reconstruction space, and determine the shortest distance from the feature point of the target object to the edge of the bridge construction side in the three-dimensional reconstruction space; and

[0057] The risk identification unit is configured to generate a bridge edge construction risk identification signal when the shortest distance is less than or equal to a preset safety threshold.

[0058] In a fourth aspect, an embodiment of the present invention provides an electronic device, which may include: a processor and a memory storing a computer program, wherein the processor is configured to implement the method as described in the first aspect or the second aspect when executing the computer program.

[0059] In a fifth aspect, an embodiment of the present invention provides a program product, comprising a computer program, wherein when the computer program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0060] In a sixth aspect, an embodiment of the present invention provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in the first aspect or the second aspect.

[0061] A method for identifying risks in bridge edge construction provided by an embodiment of the present invention obtains a plurality of position images, and the plurality of position images are collected by a camera installed on an inspection robot when the inspection robot travels to different positions along the non-construction side guardrail of the bridge; then target object detection processing is performed on the plurality of position images to obtain a detection frame and feature points of the target object in the plurality of position images; edge image segmentation processing is performed on the plurality of position images to obtain image segmentation results of the edge of the bridge construction side in the plurality of position images; and based on at least two position images among the plurality of position images, a three-dimensional reconstruction including the edge of the bridge construction side is reconstructed information; and mapping the feature points of the target object to the three-dimensional reconstruction information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space, and determining the shortest distance from the feature points of the target object to the edge of the bridge construction side in the three-dimensional reconstruction space; and when the shortest distance is less than or equal to a preset safety threshold, generating a bridge edge construction risk identification signal, so that the relative distance between the construction side edge and the target object can be quickly obtained during the inspection process, and a risk identification signal is issued in real time when the distance is less than or equal to the preset safety threshold, thereby effectively reducing the probability of safety accidents such as falling from heights and improving the safety management and control efficiency of bridge construction sites.

[0062] Other optional features and technical effects of the embodiments of the present invention are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The elements shown are not limited by the proportions shown in the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements.

[0064] Figure 1 A flow chart of a method for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0065] Figure 2 A flow chart of a method for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0066] Figure 3 A flow chart of a method for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0067] Figure 4 A flow chart of a method for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0068] Figure 5 A flow chart of a method for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0069] Figure 6 A flow chart of a method for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0070] Figure 7 A flow chart of a method for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0071] Figure 8 A flow chart of a method for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0072] Fig. 9 A flow chart of a method for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0073] Fig.10 A flow chart of a method for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0074] FIG. 11A to FIG. 11C An exemplary structural diagram of an inspection robot that can be used to implement risk identification of bridge edge construction according to an embodiment of the present invention is shown.

[0075] Fig.12 A schematic diagram showing an implementation scheme for risk identification of bridge edge construction.

[0076] Fig.13 A schematic diagram showing another implementation scheme for bridge edge construction risk identification.

[0077] Fig.14 A schematic diagram of a bridge reconstruction and expansion construction edge area to which the bridge edge construction risk identification method according to an embodiment of the present invention can be applied is shown.

[0078] Fig.15 An exemplary module diagram of a device for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0079] Fig.16 An exemplary module diagram of a device for identifying risks in bridge edge construction according to an embodiment of the present invention is shown.

[0080] Fig.17 An exemplary structural diagram of an electronic device capable of implementing the method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and drawings. Here, the exemplary implementation methods of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0082] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one some embodiments". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0083] As mentioned above, during the reconstruction, expansion or maintenance of bridges, it is often necessary to perform high-altitude operations near the edge of the bridge. Since the operation area is adjacent to the area under the bridge with a large drop, once construction workers and construction equipment get too close to the edge without safety measures, it is very easy to cause safety accidents such as falls. However, in current bridge construction, prevention often relies on manual inspections, and there is a lack of sufficiently accurate spatial distance judgment for the spatial position of the construction side edge and the location of construction workers and equipment. Especially when personnel and equipment are often moving on the construction site and the environment is complex, it is difficult to timely and accurately identify and warn of potential bridge edge construction risks. In addition, the conventional edge risk detection methods currently proposed are often difficult to apply to bridge reconstruction, expansion or maintenance construction scenarios where the construction area is constantly changing and the installation of high-altitude cameras is restricted.

[0084] In this regard, the present invention proposes a solution for identifying the risks of construction on the edge of a bridge, which can be effectively applied to the reconstruction, expansion or maintenance construction scenes of bridges, especially highway bridges, and makes full use of the inspection robot applicable to the reconstruction, expansion or maintenance construction scenes of bridges and the multiple sets of images obtained at different positions when the inspection robot and the inspection robot travel along the non-construction side guardrail, avoiding the situation where the installation of high-altitude cameras is restricted, and is suitable for the scene where the construction area may change continuously during the reconstruction, expansion or maintenance of bridges. The solution for identifying the risks of construction on the edge of a bridge proposed in the embodiment of the present invention uses the images obtained by the inspection robot camera that is difficult to use to determine the distance to the edge (the camera height is often within the height range of the construction personnel or slightly higher than that of humans), and can still effectively and efficiently determine the edge risk and generate the edge risk identification signal accordingly. In addition, the solution for identifying the risks of construction on the edge of a bridge further proposed in the embodiment of the present invention can still effectively realize the edge risk identification for target objects such as construction personnel and equipment that may be in motion. In addition, different embodiments of the present invention also provide different solutions for inspection robots with different camera configurations.

[0085] In some embodiments of the present invention, Figure 1 As shown, a method for identifying risks in bridge edge construction is provided.

[0086] In some embodiments, the method for identifying risks of construction on the edge of a bridge can be implemented by an electronic device (such as a computer) having a processor unit and a storage unit, and the electronic device can be connected to the inspection robot in communication to obtain images captured by the camera of the inspection robot walking along the non-construction side guardrail of the bridge. In some preferred embodiments, the inspection robot walking along the non-construction side guardrail of the bridge can directly implement the method of the embodiment of the present invention, for example, by including the relevant processor unit and storage unit or integrating the above-mentioned electronic device.

[0087] refer to FIG. 11A to FIG. 11C , showing multiple exemplary embodiments of an inspection robot that can be used to implement bridge edge construction risk identification according to an embodiment of the present invention or at least collect images related to bridge edge construction risk identification according to an embodiment of the present invention.

[0088] In such Fig.11A In the illustrated embodiment, the inspection robot 1 is a bottom-supported inspection robot of a corrugated guardrail, comprising a corrugated guide wheel mechanism, a drive wheel mechanism, and a housing. The corrugated guide wheel mechanism rolls along the corrugated structure of the corrugated guardrail to guide the inspection robot. The drive wheel mechanism engages the bottom edge of the corrugated guardrail from three directions, thereby not only holding the inspection robot on the corrugated guardrail by clamping, but also driving the inspection robot. In addition, the inspection robot 1 is configured to be mounted to the corrugated guardrail in a manner spaced apart from the upper edge of the corrugated guardrail. Fig.11AAs shown, the inspection robot 1 may also include an image acquisition device, i.e., a camera 2 (e.g., a single camera), which is mounted on the top of the inspection robot, for example, via a bracket, but the present invention is not limited thereto. For example, the image acquisition device, i.e., the camera 2, is directly mounted or fixed to the housing of the inspection robot. Here, although the exemplary inspection robot proposed by the inventor may have a retractable bracket to achieve the height adjustability of the image acquisition device, i.e., the camera, during normal travel, the camera is often expected to be in a low position to ensure smooth travel and avoid interference. Therefore, as described above, during the travel of the inspection robot, the camera often acquires images at a height within the range of human height or slightly above human height, such as 60cm-220cm (but multiple embodiments of the present invention are not necessarily limited to this range). Fig.11A Other features or functions of the inspection robot 1 shown may be optionally referred to, for example, patent application publication number CN 118478338 A having a common applicant, which is incorporated herein by reference in a non-contradictory manner.

[0089] In such Fig. 11B In the illustrated embodiment, the inspection robot 1 is a magnetically attracted magnetic wheel guided inspection robot for a corrugated guardrail, which may include a magnetic corrugated guide wheel mechanism, a drive wheel mechanism and a shell. The magnetic corrugated guide wheel mechanism is not only configured to roll along the corrugated structure of the corrugated guardrail to guide the inspection robot to move forward, but also to keep the inspection robot adsorbed to the corrugated guardrail by retaining magnets and to keep the inspection robot rolling correctly along the corrugated structure by the corrugated guide magnetic wheel. The drive wheel mechanism contacts the lower edge of the corrugated guardrail only on one side, thereby only playing a driving role, and does not provide any retaining function. Fig. 11B As shown, the inspection robot 1 may also include an image acquisition device, namely a camera 2 (eg, a single camera), and may refer to Fig.11A The embodiments shown are not described in detail here. Fig. 11B Other features or functions of the inspection robot 1 shown may be optionally referred to, for example, patent application publication number CN 118596114 A having a common applicant, which is incorporated herein by reference in a non-contradictory manner.

[0090] In such Fig. 11C In the embodiment shown, the inspection robot 1 is a ground-supported wheeled inspection robot for corrugated guardrails, which includes a corrugated guide wheel mechanism, a drive wheel mechanism, a ground-support wheel mechanism and a housing. Fig. 11C As shown, the inspection robot 1 may also include an image acquisition device, namely a camera 2 (eg, a single camera), and may refer to Fig.11A and Fig. 11B The embodiments shown are not described in detail here. Fig. 11COther features or functions of the inspection robot 1 shown may be optionally referred to, for example, patent publication number CN222290175U of a common applicant, which is incorporated herein by reference in a non-contradictory manner.

[0091] Combined with reference Figure 1 and Fig.12 The bridge edge construction risk identification method can be applied to the single-side construction scenario of the bridge. Figure 1 The bridge edge construction risk identification method may include the following steps S110~S160.

[0092] S110: Acquire multiple position images.

[0093] In the embodiment of the present invention, the plurality of position images (I1, I2, I3, ... I k ) (where k represents the position) is installed on the inspection robot (e.g. FIG. 11A to FIG. 11C The camera on the inspection robot (shown) collects data when the inspection robot travels to different positions along the non-construction side guardrail of the bridge.

[0094] like Fig.12 As shown, the multiple position images can be acquired by a single camera 2 on the inspection robot 1, but the present invention is not limited thereto.

[0095] In some embodiments, the inspection robot may reciprocate along the guardrail corresponding to the construction scope to acquire the multiple position images, and preferably the positions corresponding to the multiple position images are equally spaced and / or correspond to fixed positions of the guardrail. Also preferably, the multiple position images may be acquired at equal time intervals by the camera of the inspection robot with equal running speed (and / or acceleration / deceleration). In a preferred embodiment, the inspection robot remains substantially stationary (decelerating to zero or very slow speed) when acquiring multiple position images (and when acquiring optional time interval image frames). In this case, the inspection robot may move between the above-mentioned multiple positions in a periodic acceleration / deceleration motion. However, the present invention is not limited thereto; for example, the inspection robot may accelerate / decelerate for other reasons.

[0096] In the embodiment of the present invention, when describing different "position images", it means that these different position images are collected from different positions along the guardrail. Optionally, the position image may also have relevant position information (including absolute position information such as GPS position and / or road mileage position, etc., relative position information such as the interval distance of the position and / or the position sequence number of the image, etc.) or time information and / or motion information (such as speed and / or acceleration and deceleration information) that can be used to determine the position information, or the position image may be associated with such information.

[0097] S120: Perform target object detection processing on the multiple position images to obtain detection frames and feature points of the target objects in the multiple position images.

[0098] In some embodiments of the present invention, after the camera of the inspection robot collects multiple position images at different positions near the non-construction side guardrail of the bridge, the detection process of the target object can be performed on each position image to obtain the detection frame and corresponding feature points of the target object, such as the construction personnel or construction equipment in each position image. In some embodiments of the present invention, the detection frame can be used to represent the area range of the target object in the position image plane, for example, represented by {x min ,y min ,x max ,y max In some embodiments of the present invention, the feature point can be determined based on the detection frame. In a preferred embodiment, the feature point is the center point of the bottom line of the detection frame, which can, for example, roughly represent the position where the target object, especially the construction worker, touches the ground. In some embodiments, the detection frame is a roughly human-shaped frame, in which case the center position of the sole of the detected construction worker can be selected as the feature point. However, any other suitable feature points can also be thought of.

[0099] In an embodiment of the present invention, a real-time target detection algorithm such as a YOLO (You Only Look Once) series model (such as YOLOv5, YOLOv7, and YOLOv8) can be used to perform target detection on each position image. Optionally, an IoU (Intersection over Union) threshold can be set in conjunction with a confidence threshold to filter or merge detection boxes to improve the stability of the detection results. However, the present invention is not limited to the above-mentioned specific target detection algorithm.

[0100] In a preferred embodiment, the detection frame and / or feature points may be associated with an identity ID of a target object (such as a construction worker or construction equipment).

[0101] S130: Perform edge image segmentation processing on the multiple position images to obtain image segmentation results of the edge of the bridge construction side in the multiple position images.

[0102] In some embodiments of the present invention, edge image segmentation processing can be performed on the images collected by the inspection robot at different positions, so as to obtain image segmentation results of the edge of the bridge construction side. Specifically, a semantic segmentation algorithm based on pixel-level classification (such as U-Net or other deep convolutional networks) can be used in each position image to distinguish the edge of the bridge construction side from other areas, and output a corresponding segmentation mask.

[0103] In some embodiments, the image segmentation result of the edge of the bridge construction side may include the image segmentation result of the bridge construction area containing the edge (i.e., the edge area), for example, when no safety fence is set. In another embodiment, the image segmentation result of the non-bridge area can still be used to obtain the edge segmentation result. In a specific example, a safety fence is often installed on the bridge construction side (as shown in Figure 14). At this time, an image segmentation algorithm can be used to perform image segmentation on the safety fence, and the lower edge of the image segmentation area of ​​the safety fence corresponds to the edge of the bridge construction side, which also falls within the scope of the present invention.

[0104] Accordingly, in some embodiments, the image segmentation result of the bridge construction side edge will include the pixel coordinates in the corresponding position image. In some embodiments, the image segmentation result of the bridge construction side edge may also include the inferred pixel coordinates in the corresponding position image. For example, due to the occlusion of construction personnel or other objects, a part of the edge may not be displayed in the position image. At this time, the segmentation result of the edge of the occluded part can be inferred based on the pixel extension line or connection line of the edge segment if it is not occluded, but the present invention is not limited to this.

[0105] Next, the steps S140 and S150 of the bridge edge construction risk identification method are described.

[0106] S140: Reconstructing three-dimensional reconstruction space information including the construction side edge of the bridge based on at least two position images among the multiple position images.

[0107] S150: Mapping the feature points of the target object to the three-dimensional reconstruction information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space, and determining the shortest distance from the feature points of the target object to the edge of the bridge construction side in the three-dimensional reconstruction space.

[0108] In different embodiments of the present invention, the three-dimensional reconstruction and related three-dimensional position determination in the above-mentioned step S140 and step S150 can be implemented through different implementations.

[0109] In one implementation, the three-dimensional reconstruction of the above step S140 and step S150 may be implemented based on multi-frame incremental three-dimensional reconstruction, more specifically, based on incremental three-dimensional point cloud reconstruction.

[0110] In some examples of this embodiment, Figure 4As shown, the above-mentioned step S140 may include steps S410~S460: S410: extracting scene reference points for each of the multiple position images that have been acquired; S420: establishing a correspondence between the scene reference points of the multiple position images that have been acquired, and constructing a three-dimensional space structure; S430: sequentially matching the scene reference points of the newly acquired position images with the three-dimensional space structure, and updating the three-dimensional space structure; S440: mapping or updating the image segmentation result of the edge to the three-dimensional space structure to obtain three-dimensional reconstructed space information including the edge of the bridge construction side.

[0111] In the embodiment of the present invention, the scene reference point may include, for example, a road marking feature point, a safety fence feature point, or a ground identification point, but the present invention is not limited thereto.

[0112] An exemplary specific embodiment involving steps S410 - S460 will be described below.

[0113] As described above, as the camera of the inspection robot moves, it continuously collects multiple position images (I1, I2, I3, ...). Since the camera does not change its posture relative to the subject of the inspection robot when collecting images and roughly translates along the guardrail and approximates the extension direction of the road, it is possible to match the scene reference points in the acquired position images for a given scene reference point to establish a corresponding relationship, thereby reconstructing the three-dimensional point cloud structure of the scene. Furthermore, when the inspection robot enters a new position and collects a new position image, the given scene reference point in the newly collected position image can be extracted and matched, thereby updating the three-dimensional point cloud structure. In an embodiment of the present invention, any suitable method can be used to extract the above-mentioned scene reference points from each position image. In an embodiment of the present invention, the generation of the three-dimensional point cloud can be achieved by motion structure reconstruction (SfM), but the present invention is not limited to this. For example, a simultaneous localization and mapping (SLAM) algorithm can also be considered.

[0114] As an example, after the first and second position images I1, I2 are acquired (but more acquired images can also be matched), the first and second position images can be matched with reference point features, that is, scene reference points representing the same physical position in the two position images are found. As an explanation but not limitation, when the reference points of the images are matched, only some of the reference points may be matched, and as new position images are continuously acquired, some matching reference points may also be excluded. When the reference point matching is achieved, the pixel coordinates (i.e., two-dimensional coordinates) of the matched scene reference points in these position images can be subjected to three-dimensional reconstruction calculations according to the acquisition position (posture) information of the position images, including but not limited to back-projection calculations, stereo triangulation calculations, etc., to obtain the three-dimensional coordinates of these points, thereby constructing an initial three-dimensional (sparse point cloud) spatial structure. Then, based on the acquisition position (posture) information of the newly acquired position image, the three-dimensional reconstruction calculation is performed using the correspondence between the existing three-dimensional points and the corresponding two-dimensional points in the newly acquired position image, and the three-dimensional reconstruction calculations are added to the three-dimensional spatial structure. Here, people will understand that, as described in the above step S430, the three-dimensional spatial structure is continuously updated as the position images are continuously acquired.

[0115] In a further embodiment, the edge pixels obtained by the image segmentation process in step S130 can be similarly projected into the three-dimensional point cloud structure to determine the three-dimensional position C of the edge in the three-dimensional coordinate system. edge . The mapping or updating described in the above step S440 can be interpreted broadly. For example, the three-dimensional position of the edge can be obtained based on the projection result. Alternatively, the original three-dimensional spatial structure already contains the original position of the edge. At this time, the three-dimensional position obtained by the new projection or the original three-dimensional position can be compared. If it is within the error range, the original three-dimensional position or the new projection three-dimensional position or their mean or weighted mean can be selected as the three-dimensional position of the edge, which all falls within the scope of the present invention.

[0116] Subsequently, in step S150 , the feature points of the target object may be mapped into the three-dimensional space structure to determine the positions of the feature points in the three-dimensional space structure.

[0117] For example, reference Figure 5 As shown, more specifically, step S150 may include the following steps: S510: obtaining pixel coordinates of feature points of the target object in at least one position image of multiple position images; S520: determining the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space based on the spatial transformation relationship between the pixel coordinates or based on the mapping relationship between the pixel coordinates and the depth.

[0118] In a preferred embodiment, when mapping feature points, the motion displacement corresponding to the target may be optionally considered, and motion displacement compensation may be performed during mapping based on the motion situation. However, the embodiment in the first implementation mode described above may not be limited thereto, that is, motion displacement compensation may not be performed when performing the above steps S510 and S520.

[0119] Accordingly, if Figure 2 and Figure 3 As shown, the bridge edge construction risk identification method may further include the following steps S210-S230.

[0120] S210: Acquire image frames at least two time intervals at each different position.

[0121] In this embodiment, each of the time-interval image frames includes a plurality of images acquired at the same location at a preset time interval. In some preferred embodiments, a pair of image frames at a predetermined time interval may be acquired at each location, such as a first image frame and a second image frame (I k,1 , I k,2 ).

[0122] In some embodiments, step S210 may be implemented in an integrated manner with step S110, so that the position image I obtained in step S110 k It can be used as one of the image frames of the at least two time intervals, such as the first image frame I of the above-mentioned image frame pair. k,1 , so that other image frames, such as the second image frame I, can be acquired at a predetermined time interval after the position image is acquired. k,2 For ease of description, the position image I k The first image frame and the second image frame with a time interval (I k,1 , I k,2 ) Different expressions are still used below, regardless of whether the position image constitutes an image frame of one of the time intervals.

[0123] S220: Perform target detection processing on image frames at least in two time intervals to obtain position information of the target object in the image frames at least in two time intervals.

[0124] In these embodiments, a target detection algorithm may be used to detect image frames with time intervals, such as the first image frame and the second image frame (I k,1 , I k,2 ) Perform target detection respectively to obtain the position information of the target object (such as construction personnel or equipment) in the two frames of images. In a preferred embodiment, the position information of the feature points described in step S120 can be used to represent the position information of the target object.

[0125] In these embodiments, the target detection process of step S220 may adopt the same method as step S120, which will not be described in detail here, but the present invention is not limited thereto.

[0126] S230: Determine the motion state of the target object based on the position information.

[0127] In step S230, by comparing the target object in the first image frame I k,1 and the second image frame I k,2 The position difference of the target object in the image, especially the position difference of the feature point, can be used to determine whether the target object has moved significantly during the preset time interval and determine the movement state.

[0128] More specifically, in some embodiments of the present invention, step S230 may include the following steps: S310: calculating the pixel displacement between the feature point positions of the target object in the first image and the second image; and S320: when the pixel displacement is greater than a preset threshold, determining that the target object is in motion.

[0129] In step S230, by comparing the target object in the first image frame I k,1 and the second image frame I k,2 The position of the feature points in the image can be used to determine whether the target object has moved within the preset time interval.

[0130] As mentioned above, assuming that the bottom center point of the detection frame in the time interval image frame is still used as the feature point to determine the position difference of the feature point, the first image frame I k,1 and the second image frame I k,2 The pixel coordinates can be expressed as: .

[0131] At this time, if , If the target object is less than or equal to the preset threshold (e.g., n pixels), it is determined that the target object has moved significantly, and the pixel displacement or pixel speed can be estimated to perform motion displacement compensation on the three-dimensional position of the feature point of the target object in the position image. That is, at this time, step S150 will include: performing motion displacement compensation on the three-dimensional position of the feature point of the target object based on the motion state of the target object. This will be further described below. If it is less than or equal to the preset threshold, it is determined that the target is approximately stationary, and no motion displacement compensation is performed.

[0132] In a specific embodiment of the present invention, a three-dimensional position determination process will be described when the target object is in a substantially stationary state (i.e., the pixel displacement calculated in step S310 is not greater than a preset threshold ε). The three-dimensional position determination process is also applicable to a solution that does not consider the motion of the target object.

[0133] In this case, since the target object does not move significantly between the time interval image frames (Ik,1, Ik,2), it means that the target object remains relatively still at the current observation time. At this time, the spatial relationship between images at different positions can be directly used to determine the three-dimensional position without considering the position offset caused by the target movement.

[0134] Specifically, after obtaining the image I of the target object at the current position k The feature point position p in k After (for example, the coordinates of the bottom center point of the detection frame obtained by target detection), the adjacent position image (such as I k-1 ) to determine the three-dimensional position, and find the position p of the feature point corresponding to the same target object in the adjacent position image k-1 .

[0135] After the correspondence between the feature points of the target object between the position images is established, since the target object remains stationary, the corresponding feature points in the two position images actually represent the same physical position in the three-dimensional space. At this point, based on the obtained three-dimensional reconstruction space information (see step S140), we know that the position image I k and I k-1 Using this information, the three-dimensional position of the feature points of the target object in three-dimensional space can be calculated by stereo triangulation method: object = Triangulate(p k-1 , p k , T k-1 , T k , K), where P is the three-dimensional position in the three-dimensional space, p is the pixel position in the position image (or called the two-dimensional position), T is the position (posture) information of the position image (camera), and K is the camera parameter.

[0136] In an alternative embodiment, the three-dimensional position of the target object can also be determined by a back-projection algorithm. Specifically, the depth value corresponding to the feature point can be determined by query or interpolation (for example, by interpolation of adjacent fixed points) based on the pixel coordinates of the feature point of the target object in the current position image, using the depth information in the three-dimensional reconstruction space information, and based on the depth value of the determined feature point and the position (posture) information of the position image (or camera), the three-dimensional position of the target object (that is, the three-dimensional position of the feature point) is obtained by back-projection. As an explanation but not limitation, this alternative embodiment can not only be used to determine the position of the target object in a roughly stationary state, but can also be applied to a solution that does not consider the movement of the target object.

[0137] When the target object is in motion (ie, the pixel displacement calculated in step S310 is greater than a preset threshold ε), motion displacement compensation may be performed on the feature point position of the target object.

[0138] For example, reference Figure 6 As shown, more specifically, when performing motion displacement compensation, step S150 may include the following steps: S610: obtaining pixel coordinates of the feature point of the target object in at least one position image of multiple position images; S620: determining the motion speed of the target object based on the pixel difference and time interval of the feature point of the target object in image frames of at least two time intervals, and performing motion displacement compensation on the pixel coordinates according to the motion speed; S630: determining the three-dimensional position of the feature point of the target object in the three-dimensional reconstruction space based on the spatial transformation relationship between the compensated pixel coordinates or based on the mapping relationship between the compensated pixel coordinates and the depth. When the target object is in motion (i.e., the pixel displacement calculated in step S310 is greater than the preset threshold ε), the motion displacement compensation for the feature point position of the target object is preferably performed on the feature point position p in the previous position image. k-1 In this embodiment, when it is determined that the target object at the current position is in motion, the position p of the feature point in the previous position image is compensated. k-1 To perform motion compensation, we can also consider the feature point position p in the current position image. k Perform motion compensation (the difference is that it is compensated in the opposite direction).

[0139] More specifically, the motion speed of the target object can be calculated based on the position difference of the feature points in the time interval image frames: Δp = p k,2 -p k,1 , v = Δp / Δt, where p k,2 and p k,1 are pixel positions of feature points of the target object in the first image frame and the second image frame respectively, and Δt is a preset time interval.

[0140] Then, when determining the three-dimensional position of the target object, it is necessary to consider the movement of the target object from the shooting time of the adjacent (previous) position image to the shooting time of the current position image. In a preferred embodiment of the present invention, the feature point position can be compensated based on the linear motion assumption: according to the calculated motion speed v and the time difference ΔT of the image shooting time (for example, it can be determined based on the position information and / or speed information and / or image acquisition time information of the two position images), the motion displacement compensation amount ΔP = v×ΔT of the feature point position of the adjacent (previous) position image is calculated, and the feature point position in the current position image is adjusted accordingly, so that it corresponds to the same position as the feature point assumption in the adjacent position image.

[0141] After completing motion compensation, the same triangulation method as in the static state can be used to determine the three-dimensional position of the target object using the compensated feature point positions.

[0142] Similarly, in an alternative embodiment, the three-dimensional position of the target object can also be determined by a back-projection algorithm based on the compensated pixel position. Although as described above, in some embodiments, this implementation can be applicable to a solution that does not consider the motion of the target object; however, in an optional embodiment, a back-projection algorithm can be used for the compensated pixel position. For example, when the feature points in the current position image cannot well characterize the contact position between the target object and the bridge deck (for example, the lower body of the current construction worker is blocked by the foreground), motion displacement compensation can be used to perform a back-projection operation based on the compensated pixel coordinates of the feature points of the previous position image, thereby approximately determining the current bridge deck position of the target object, which falls within the scope of the present invention.

[0143] Accordingly, after obtaining the three-dimensional position of the feature point of the target object in the three-dimensional reconstruction space, the shortest distance from the feature point of the target object to the edge of the bridge construction side in the three-dimensional reconstruction space can be determined.

[0144] Specifically, for example, in the three-dimensional position P of the feature point object and the 3D position C of the edge (line segment) edge After that, the shortest distance d from the feature point to the edge of the bridge construction side can be determined through the minimum distance search calculation.

[0145] Right now: .

[0146] In another embodiment, the three-dimensional reconstruction of the above step S140 and step S150 can be achieved by implementing three-dimensional depth reconstruction based on dual-position images. More specifically, in this embodiment, the three-dimensional reconstruction involves determining the moving binocular-like stereo depth information. Therefore, unlike the aforementioned incremental three-dimensional reconstruction, this embodiment uses the position image pairs obtained by the inspection robot at different positions (especially at adjacent positions) to form a spatial stereo baseline, and directly obtains the scene depth information by stereo matching.

[0147] In some examples of this embodiment, Figure 7 As shown, the above step S140 may include steps S710 to S730.

[0148] S710: Select two position images from a plurality of position images to form a spatial stereoscopic image pair.

[0149] S720: Perform stereo matching calculation on the spatial stereo image pair to obtain scene depth information.

[0150] S730: Obtain depth information of the edge of the bridge construction side according to the image segmentation result of the edge in the spatial stereo image pair and the scene depth information.

[0151] In a specific embodiment of the present invention, the inspection robot moves forward in the direction of the guardrail to collect images during movement. Since the camera is fixedly mounted on the inspection robot, the posture of the camera relative to the inspection robot remains unchanged. Therefore, two position images separated by a certain distance (for example, two adjacent position images) can be selected to form a spatial stereo image pair, and the distance between the two positions forms a spatial baseline, which is similar to the baseline in binocular stereo vision, and is used for subsequent stereo matching and depth calculation.

[0152] An exemplary specific embodiment involving steps S710 - S730 will be described in detail below.

[0153] As the inspection robot moves along the guardrail, adjacent position images I can be selected from the multiple position images obtained. k and I k-1 As a spatial stereo image pair. Since the camera is fixedly installed on the inspection robot, its posture relative to the robot body remains unchanged, and the robot generally moves along the direction of the guardrail, which makes the position image I k and I k-1 There is a relatively fixed spatial transformation relationship between them.

[0154] After obtaining the spatial stereo image pair, a position transformation operation can be performed according to the relative position (posture) difference between the two positions, which is a stereo matching operation. After determining the relative position and posture transformation relationship, a stereo matching operation is performed on the spatial stereo image pair. Through the stereo matching algorithm, the disparity between corresponding pixels in the image pair can be calculated, and then the depth information of the scene can be obtained. In this way, a stereogram with depth of field information can be formed. In an embodiment of the present invention, the scene depth information determined based on binocular stereo vision is interpreted broadly, and also includes disparity information, thereby forming a binocular stereo reconstruction. In an embodiment of the present invention, the above-mentioned stereo matching operation can exclude target objects detected by, for example, a target detection algorithm, or perform stereo matching operations only based on fixed features in the two images, which will not be repeated here.

[0155] In a preferred embodiment, the edge is often fixed in a short time, so the segmentation result of the edge in the two position images (the pixel position of the line segment) or a part thereof can also be used in the above stereo matching calculation, so the scene depth information obtained above will contain the three-dimensional position (including depth (depth of field)) of the edge or a part thereof. However, the present invention is not limited to this. For example, the depth information of the edge can be combined with the scene depth of field information determined above, and determined in a separate step according to the segmentation result of the edge, which falls within the scope of the present invention.

[0156] Subsequently, the depth of the feature point of the target object may be determined in step S150 according to the pixel position of the target object in the two position images in combination with the determined scene depth information.

[0157] For example, reference Figure 8 As shown, more specifically, step S150 may include the following steps: S810: obtaining pixel coordinates of feature points of the target object in the spatial stereo image pair; S820: determining the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space based on the pixel coordinates and scene depth information.

[0158] In some embodiments, the pixel positions of the feature points of the target object in the two position images can be inspected and calculated in the same stereo matching motion manner to determine the three-dimensional position (including the depth position) of the feature points, and then associated with the scene depth information determined above. As another embodiment, the corresponding position of the feature points of the target object in the stereogram with depth of field information can be directly determined based on the pixel positions of the feature points of the target object in the two position images, which falls within the scope of the present invention.

[0159] In this implementation manner, for matters not covered, reference may be made to the relevant contents of binocular stereoscopic display, which will not be elaborated here.

[0160] In a preferred embodiment, when mapping feature points, the motion displacement of the target may be optionally considered, and motion displacement compensation may be performed during mapping based on the motion situation. Figure 2 and Figure 3 The bridge edge construction risk identification method may further include the following steps S210~S230.

[0161] S210: Acquire image frames at least two time intervals at each different position.

[0162] In this embodiment, each of the time-interval image frames includes a plurality of images acquired at the same location at a preset time interval. In some preferred embodiments, a pair of image frames at a predetermined time interval may be acquired at each location, such as a first image frame and a second image frame (I k,1 , I k,2 ).

[0163] In some embodiments, step S210 may be implemented in an integrated manner with step S110, so that the position image I obtained in step S110 k It can be used as one of the image frames of the at least two time intervals, such as the first image frame I of the above-mentioned image frame pair. k,1 , so that other image frames, such as the second image frame I, can be acquired at a predetermined time interval after the position image is acquired.k,2 For ease of description, the position image I k The first image frame and the second image frame with a time interval (I k,1 , I k,2 ) Different expressions are still used below, regardless of whether the position image constitutes an image frame of one of the time intervals.

[0164] S220: Perform target detection processing on image frames at least in two time intervals to obtain position information of the target object in the image frames at least in two time intervals.

[0165] In these embodiments, a target detection algorithm may be used to detect image frames with time intervals, such as the first image frame and the second image frame (I k,1 , I k,2 ) Perform target detection respectively to obtain the position information of the target object (such as construction personnel or equipment) in the two frames of images. In a preferred embodiment, the position information of the feature points described in step S120 can be used to represent the position information of the target object.

[0166] In these embodiments, the target detection process of step S220 may adopt the same method as step S120, which will not be described in detail here, but the present invention is not limited thereto.

[0167] S230: Determine the motion state of the target object based on the position information.

[0168] In step S230, by comparing the target object in the first image frame I k,1 and the second image frame I k,2 The position difference of the target object in the image, especially the position difference of the feature point, can be used to determine whether the target object has moved significantly during the preset time interval and determine the movement state.

[0169] More specifically, in some embodiments of the present invention, step S230 may include the following steps: S310: calculating the pixel displacement between the feature point positions of the target object in the first image and the second image; and S320: when the pixel displacement is greater than a preset threshold, determining that the target object is in motion.

[0170] In step S230, by comparing the target object in the first image frame I k,1 and the second image frame I k,2 The position of the feature points in the image can be used to determine whether the target object has moved within the preset time interval.

[0171] As mentioned above, assuming that the bottom center point of the detection frame in the time interval image frame is still used as the feature point to determine the position difference of the feature point, the first image frame I k,1 and the second image frame Ik,2 The pixel coordinates can be expressed as: .

[0172] At this time, if , If the target object is less than or equal to the preset threshold (e.g., n pixels), it is determined that the target object has moved significantly, and the pixel displacement or pixel speed can be estimated to perform motion displacement compensation on the three-dimensional position of the feature point of the target object in the position image. That is, at this time, step S150 will include: performing motion displacement compensation on the three-dimensional position of the feature point of the target object based on the motion state of the target object. This will be further described below. If it is less than or equal to the preset threshold, it is determined that the target is approximately stationary, and no motion displacement compensation is performed.

[0173] When no motion compensation is performed, the determination of the three-dimensional position (including depth) of the target object can refer to the above description of Figure 8 Description.

[0174] In this other embodiment, for example, referring to Fig. 9 As shown, when performing motion displacement compensation, step S150 may include the following steps: S910: obtaining pixel coordinates of the feature points of the target object in the spatial stereo image pair; S920: determining the motion speed of the target object based on the pixel difference and time interval of the image frames of the target object in at least two time intervals, and performing motion displacement compensation on the pixel coordinates according to the motion speed. S930: determining the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space based on the compensated pixel coordinates and scene depth information.

[0175] Specifically, first obtain the pixel coordinates pk and pk-1 of the feature points of the target object in the spatial stereo image pair (Ik, Ik-1). When the target object is in motion, the pixel coordinates of these two positions do not correspond to the spatial positions at the same time, so motion compensation is required.

[0176] More specifically, the motion speed of the target object can be calculated based on the position difference of the feature points in the time interval image frames: Δp = p k,2 -p k,1 , v = Δp / Δt, where p k,2 and p k,1 are pixel positions of feature points of the target object in the first image frame and the second image frame respectively, and Δt is a preset time interval.

[0177] In some embodiments, the feature points in one of the positions can be fixed, and the image of another position k or k-1 can be compensated according to the motion speed determined at position k. In different embodiments, the motion speed of the previous position can also be calculated. In other embodiments, the motion speeds of both positions can also be determined and averaged, which all fall within the scope of the present invention.

[0178] Motion displacement compensation can be based on the calculated motion speed v and the time difference ΔT at the time of image capture (for example, it can be determined based on the position information and / or speed information and / or image acquisition time information of the two position images), calculate the motion displacement compensation amount ΔP = v×ΔT of the feature point position of the position image to be compensated, and adjust the position of the feature point in the position image to be compensated accordingly. Then, the pixel position of the feature point after compensation adjustment is calculated as in the previous reference. Figure 8 The method described above determines the three-dimensional position (including depth) of the target object, which will not be described in detail here.

[0179] Specifically, for example, in the three-dimensional position P of the feature point object and the 3D position C of the edge (line segment) edge After that, the shortest distance d from the feature point to the edge of the bridge construction side can be determined similarly through the minimum distance search calculation.

[0180] Right now: .

[0181] S160: When the shortest distance is less than or equal to a preset safety threshold, a bridge edge construction risk identification signal is generated.

[0182] In different embodiments, the risk identification signal may be in different forms, and there may be different response results for the risk identification signal.

[0183] In some embodiments, the risk identification signal will trigger an additional safety belt detection, and trigger an alarm when the target object (such as a construction worker) is not wearing a safety belt. In other embodiments, the method may also include a safety pending detection, and trigger an alarm when it is detected that the safety belt is not worn and a risk identification signal is generated.

[0184] In some embodiments, the risk identification signal will trigger additional safety guardrail detection or trigger the acquisition of previously generated safety guardrail detection results (for example, obtained in step S130). When there is no safety guardrail or the safety guardrail is damaged, it may optionally trigger a seat belt detection or directly trigger an alarm.

[0185] The above contents all fall within the scope of the present invention.

[0186] In another embodiment of the present invention, Fig.10As shown, a method for identifying risks in bridge edge construction is provided, which may at least include the following S1010~S1060.

[0187] S1010: Acquire binocular image pairs at multiple positions.

[0188] The binocular image pairs at the multiple positions are synchronously collected by a pair of cameras installed on the inspection robot when the inspection robot moves to different positions along the non-construction side guardrail of the bridge.

[0189] S1020: Perform target object detection processing on at least one image in the binocular image pair at each position to obtain a detection frame of the target object and its feature points.

[0190] S1030: Perform edge image segmentation processing on at least one image in the binocular image pair at each position to obtain an image segmentation result of the edge of the bridge construction side.

[0191] S1040: Perform stereo matching calculation on the binocular image pair at each position to obtain scene depth information corresponding to the position.

[0192] The scene depth information includes the depth information of the edge of the bridge construction side.

[0193] S1050: Determine the depth information of the feature point of the target object according to the pixel coordinates of the feature point of the target object at the position and the scene depth information, and determine the shortest distance from the feature point of the target object to the edge of the bridge construction side according to the depth information of the edge of the bridge construction side and the depth information of the feature point.

[0194] S1060: When the shortest distance is less than or equal to a preset safety threshold, a bridge edge construction risk identification signal is generated.

[0195] In this embodiment, a method similar to the above is provided. Figures 7 and 8 The bridge edge construction risk identification method based on dual purpose of the embodiment shown in the figure is different in that: Fig.10 The bridge edge construction risk identification method of the embodiment shown is based on the real entity binocular camera to obtain (such as Fig.13 Schematically shown), rather than Figures 7 and 8 The binocular-like solution obtained by multiple position acquisition (and optional displacement compensation) is shown. Fig.10 In the embodiment shown, displacement compensation may not be used, although Fig.10 The embodiment shown requires the use of a more expensive inspection robot configuration.

[0196] like Fig.13As shown, the inspection robot 1 may include a pair of cameras 2 and 3, the pair of cameras including an upper camera and a lower camera, the optical axes of the upper camera and the lower camera are parallel, and in the vertical direction, for example, they may have a preset baseline distance of 20 to 50 cm. Thus, in step S1010, a binocular image pair may be acquired at each position. Here, it will be understood that Fig.10 The method of the illustrated embodiment can be implemented by any suitable inspection robot, such as Figures 11A-11C The ones shown, differ in that an additional camera is added to form a binocular camera.

[0197] The features of steps S1020 to S1060 can be referred to above, especially steps S1040 and S1050 can be referred to in conjunction with Figures 7 and 8 In the related description of the embodiment described above, it will be understood that the binocular image pair I acquired in one position j,1 ,I j,2 will roughly correspond to the above Figures 7 and 8 The image I of the two positions of the feature k , I k-1 The image frames that do not require time intervals are not described here.

[0198] It should be noted that the features in the various embodiments described in the present invention can be combined or replaced with each other without conflict. For example, the binocular image pair processing method (such as stereo matching, depth calculation, etc.) described in this embodiment can refer to the relevant description in the previous embodiment; similarly, some features in the previous embodiment (such as target detection method, edge segmentation processing, three-dimensional position determination, etc.) can also be applied to this embodiment. For example, Figures 7 and 8 The stereo matching principle, three-dimensional position calculation method and other technical features in the above embodiment can also be applied to Fig.10 In the illustrated embodiment, the reverse is true.

[0199] In some embodiments of the present invention, Fig.15 As shown, a bridge edge construction risk identification device 1500 is also provided. The bridge edge construction risk identification device 1500 may include: an image acquisition unit 1510, a detection unit 1520, a segmentation unit 1530, a three-dimensional reconstruction unit 1540, a mapping unit 1550 and a risk identification unit 1560.

[0200] In other embodiments of the present invention, Fig.16 As shown, a bridge edge construction risk identification device 1600 is also provided, which may include: an image acquisition unit 1610, a detection unit 1620, a segmentation unit 1630, a three-dimensional reconstruction unit 1640, a mapping unit 1650 and a risk identification unit 1660.

[0201] In these embodiments, the image acquisition unit 1610 is configured to acquire binocular image pairs at multiple positions, and the binocular image pairs at multiple positions are synchronously collected by a pair of cameras installed on the inspection robot when the inspection robot travels to different positions along the non-construction side guardrail of the bridge.

[0202] In these embodiments, the detection unit 1620 is configured to perform target object detection processing on at least one image in the binocular image pair at each position to obtain a detection frame of the target object and its feature points.

[0203] In these embodiments, the segmentation unit 1630 is configured to perform edge image segmentation processing on at least one image in the binocular image pair at each position to obtain an image segmentation result of the edge of the bridge construction side.

[0204] In these embodiments, the 3D reconstruction unit 1640 is configured to perform stereo matching calculation on the binocular image pair at each position to obtain scene depth information corresponding to the position, wherein the scene depth information includes depth information of the edge of the bridge construction side.

[0205] In these embodiments, the mapping unit 1650 is configured to determine the depth information of the feature point of the target object based on the pixel coordinates of the feature point of the target object at the position and the scene depth information, and to determine the shortest distance from the feature point of the target object to the edge of the bridge construction side based on the depth information of the bridge construction side edge and the depth information of the feature point.

[0206] In these embodiments, the risk identification unit 1660 is configured to generate a bridge edge construction risk identification signal when the shortest distance is less than or equal to a preset safety threshold.

[0207] In the embodiments of the present invention, the device embodiments may refer to the method embodiments and may implement the steps described in the method embodiments, and the features of the device embodiments may be incorporated into the method embodiments in a non-contradictory manner, and vice versa. In addition, the features of one device embodiment may be incorporated into another device embodiment in a non-contradictory manner, and vice versa.

[0208] In some embodiments of the present invention, an electronic device is provided, which includes a processor and a memory storing a computer program, wherein the processor is configured to implement a method of any embodiment of the present invention when running the computer program.

[0209] Fig.17A schematic diagram of an electronic device 1700 that can be used to implement a method or implement an embodiment of the present invention is shown. In some embodiments, the number of electronic devices may be more or less than the number shown. In some embodiments, a single or multiple electronic devices may be used for implementation. In some embodiments, cloud or distributed electronic devices may also be used for implementation.

[0210] like Fig.17 As shown, the electronic device 1700 includes a processor 1701 and a memory 1702. The processor is used to execute programs stored in the memory, and these programs can implement the methods, steps or functions described in the above embodiments when executed by the computer. The processor 1701 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. The processor 1701 and the memory 1702 are interconnected through a bus 1703. The bus 1703 can also be connected to an input / output (I / O) interface, etc.

[0211] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0212] Although not shown, in an embodiment of the present invention, a program product is provided, including a computer program, and when the computer program is executed by a processor, the method of any one of the embodiments of the present invention is implemented.

[0213] Although not shown, in an embodiment of the present invention, a storage medium is provided, the storage medium storing a computer program, the computer program being configured to implement any method of the embodiments of the present invention when executed.

[0214] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0215] The methods, programs, systems, devices, etc. of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0216] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.

[0217] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0218] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for identifying risks in bridge edge construction, characterized in that: include: Acquire multiple position images, wherein the multiple position images are collected by a camera installed on the inspection robot when the inspection robot travels to different positions along the non-construction side guardrail of the bridge, wherein image frames of at least two time intervals are acquired at each of the different positions; Performing target object detection processing on the multiple position images to obtain detection frames and feature points of the target objects in the multiple position images; Performing target object detection processing on the image frames at least two time intervals to obtain position information of the target object in the image frames at least two time intervals; determining a motion state of the target object based on the position information; Performing edge image segmentation processing on the multiple position images to obtain image segmentation results of the edge of the bridge construction side in the multiple position images; Reconstructing three-dimensional reconstruction space information including the construction side edge of the bridge based on at least two of the multiple position images; Mapping the feature points of the target object to the three-dimensional reconstruction space information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space, and when the target object is in motion, comprising: performing motion displacement compensation on the three-dimensional position of the feature points of the target object based on the motion state of the target object; Determining the shortest distance from the feature point of the target object to the edge of the bridge construction side in the three-dimensional reconstruction space; and When the shortest distance is less than or equal to a preset safety threshold, a bridge edge construction risk identification signal is generated.

2. The bridge edge construction risk identification method according to claim 1 is characterized in that: The reconstructing three-dimensional reconstruction space information including the construction side edge of the bridge based on at least two position images among the multiple position images comprises: Extracting a scene reference point from each of the plurality of acquired location images, wherein the scene reference point includes a road marking feature point, a safety fence feature point, or a ground identification point; Establishing the correspondence between scene reference points of multiple acquired position images to construct a three-dimensional space structure; sequentially matching the scene reference points of the newly acquired position image with the three-dimensional space structure to update the three-dimensional space structure; Mapping or updating the image segmentation result of the edge into the three-dimensional space structure to obtain three-dimensional reconstructed space information including the edge of the bridge construction side; Mapping the feature points of the target object to the three-dimensional reconstruction space information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space includes: Acquire pixel coordinates of a feature point of the target object in at least one position image of a plurality of position images; Based on the spatial transformation relationship between pixel coordinates or based on the mapping relationship between pixel coordinates and depth, the three-dimensional position of the feature point of the target object in the three-dimensional reconstruction space is determined.

3. The bridge edge construction risk identification method according to claim 2 is characterized in that: When the target object is in motion, mapping the feature points of the target object to the three-dimensional reconstruction space information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space includes: Acquire pixel coordinates of a feature point of the target object in at least one position image of a plurality of position images; Determine the motion speed of the target object based on the pixel difference of the feature point of the target object in the image frames of the at least two time intervals and the time interval, and perform motion displacement compensation on the pixel coordinates according to the motion speed; Based on the spatial transformation relationship between the compensated pixel coordinates or based on the mapping relationship between the compensated pixel coordinates and the depth, the three-dimensional position of the feature point of the target object in the three-dimensional reconstruction space is determined.

4. The bridge edge construction risk identification method according to claim 1 is characterized in that: The reconstructing three-dimensional reconstruction space information including the construction side edge of the bridge based on at least two position images among the multiple position images comprises: Selecting two position images from the plurality of position images to form a spatial stereoscopic image pair; Performing stereo matching calculation on the spatial stereo image pair to obtain scene depth information; Obtaining depth information of the edge of the bridge construction side according to the image segmentation result of the edge in the spatial stereo image pair and combining the scene depth information; Mapping the feature points of the target object to the three-dimensional reconstruction space information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space includes: Obtain pixel coordinates of feature points of the target object in the spatial stereo image pair; Based on the pixel coordinates and the scene depth information, a three-dimensional position of the feature point of the target object in a three-dimensional reconstruction space is determined.

5. The bridge edge construction risk identification method according to claim 4 is characterized in that: When the target object is in motion, mapping the feature points of the target object to the three-dimensional reconstruction space information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space includes: Obtain pixel coordinates of feature points of the target object in the spatial stereo image pair; Determine the motion speed of the target object based on the pixel difference of the target object in the image frames of the at least two time intervals and the time interval, and perform motion displacement compensation on the pixel coordinates according to the motion speed; Based on the compensated pixel coordinates and the scene depth information, the three-dimensional position of the feature point of the target object in the three-dimensional reconstruction space is determined.

6. A device for identifying risks in bridge edge construction, characterized in that: include: An image acquisition unit is configured to acquire a plurality of position images, wherein the plurality of position images are acquired by a camera installed on the inspection robot when the inspection robot travels to different positions along the non-construction side guardrail of the bridge, wherein image frames of at least two time intervals are acquired at each of the different positions; a detection unit configured to perform target object detection processing on the multiple position images to obtain a detection frame and feature points of the target object in the multiple position images, and further configured to perform target object detection processing on the image frames of the at least two time intervals to obtain position information of the target object in the image frames of the at least two time intervals, and determine a motion state of the target object based on the position information; a segmentation unit configured to perform edge image segmentation processing on the plurality of position images to obtain image segmentation results of the edge of the bridge construction side in the plurality of position images; A three-dimensional reconstruction unit, configured to reconstruct three-dimensional reconstruction space information including the edge of the bridge construction side based on at least two position images of the plurality of position images; a mapping unit configured to map the feature points of the target object to the three-dimensional reconstruction space information to obtain the three-dimensional position of the feature points of the target object in the three-dimensional reconstruction space, and when the target object is in motion, comprising: performing motion displacement compensation on the three-dimensional position of the feature points of the target object based on the motion state of the target object, and determining the shortest distance from the feature points of the target object to the edge of the bridge construction side in the three-dimensional reconstruction space; and The risk identification unit is configured to generate a bridge edge construction risk identification signal when the shortest distance is less than or equal to a preset safety threshold.

7. An electronic device, characterized in that: include: A processor and a memory storing a computer program, wherein the processor is configured to implement the method according to any one of claims 1 to 5 when executing the computer program.

8. A program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images