Aerial work platform, image processing method, anti-collision control method and system thereof

Abstract

The embodiments of the present application provide an aerial work platform and its image processing method, anti-collision method and system, processor and machine-readable storage medium. The image processing method includes: determining a plurality of first points of interest in a first area corresponding to the overlapping area of shooting in a first image taken by a side camera, and a plurality of second points of interest corresponding to the plurality of first points of interest in a second image taken by a top camera, wherein the plurality of first points of interest and the second points of interest form a first grid and a second grid respectively; according to the correspondence between the first triangle in the first grid and the second triangle in the second grid, associating the pixel points in the first area with the pixel points in the second area; and assigning the color value of the pixel points in the first area to the corresponding pixel points in the second area. Through the above technical solution, the images of the side camera and the top camera can be fused to provide a more optimized visual interface for the operator.

Description

Technical Field

[0001] The present application relates to the field of aerial work platform safety control, and specifically to an image processing method, an anti-collision control method for an aerial work platform, a processor, an anti-collision control system, an aerial work platform, and a machine-readable storage medium. Background Art

[0002] Aerial work platforms (Aerial Work Platforms) are mobile products used for aerial work, equipment installation, maintenance, and other high-altitude operations across various industries. For example, traditional Aerial Work Platform products include scissor-type Aerial Work Platforms, vehicle-mounted Aerial Work Platforms, articulated boom Aerial Work Platforms, self-propelled Aerial Work Platforms, aluminum alloy Aerial Work Platforms, and cylinder-type Aerial Work Platforms.

[0003] When operating an aerial work platform to move toward a target work point, workers are unable to gain a global understanding of the platform's surroundings because their line of sight is limited to a single direction. Therefore, it is crucial to construct an anti-collision control system with surrounding environmental monitoring capabilities. This not only helps workers understand the surrounding aerial work environment, but also allows them to determine whether the work platform will collide with obstacles and subsequently implement anti-collision measures. Existing anti-collision control systems use radar sensors and / or visual sensors to detect the location of obstacles and implement corresponding anti-collision measures based on the detected obstacle locations. However, this anti-collision control system does not provide workers with a convenient visual interface for the work platform's surroundings, and the anti-collision actions performed based on the obstacle location are less comfortable. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, an anti-collision control method for an aerial work platform, a processor, an anti-collision control system, an aerial work platform, and a machine-readable storage medium.

[0005] To achieve the above objectives, the present application provides, in a first aspect, an image processing method for an aerial work platform. The aerial work platform includes a work platform, and the work platform is provided with a side camera for capturing a side environmental image of the work platform and a top camera for capturing an environmental image of the work platform from a top-down angle. The side camera and the top camera have an overlapping shooting area. The image processing method includes:

[0006] Determine a plurality of first points of interest in a first area corresponding to the overlapping area in a first image captured by the side camera, and determine a plurality of second points of interest corresponding to the plurality of first points of interest in a second area corresponding to the overlapping area in a second image captured by the top camera, wherein the plurality of first points of interest form a first mesh consisting of a plurality of first triangles with the first points of interest as vertices, and the plurality of second points of interest form a second mesh consisting of a plurality of second triangles with the second points of interest as vertices;

[0007] Associating pixels in the first area with pixels in the second area based on a correspondence between a first triangle in the first grid and a second triangle in the second grid; and

[0008] Assign the color values of the pixels in the first area to the corresponding pixels in the second area.

[0009] In the embodiment of the present application, associating pixels in the first area with pixels in the second area based on a correspondence between a first triangle in the first grid and a second triangle in the second grid includes:

[0010] For any first pixel point other than the first point of interest in the first area, determining a first target triangle where the first pixel point is located according to the pixel coordinates of the first pixel point;

[0011] Determine a weight according to the pixel coordinates of the first pixel point and the pixel coordinates of the three vertices of the first target triangle;

[0012] determining a second target triangle in the second image corresponding to the first target triangle;

[0013] Determine a second pixel point corresponding to the first pixel point in the second image according to the weight and the pixel coordinates of the three vertices of the second target triangle;

[0014] Assigning the color value of a pixel in the first area to a corresponding pixel in the second area includes: assigning the color value of the first pixel to the second pixel.

[0015] In the embodiment of the present application, determining the weight according to the pixel coordinates of the first pixel point and the pixel coordinates of the three vertices of the first target triangle includes: determining the weight according to formula (1):

[0016]

[0017] Among them, x p ,y p are the horizontal and vertical coordinates of the first pixel, respectively, A ,y A are the horizontal and vertical coordinates of the first vertex of the first target triangle, respectively, B ,yB are the horizontal and vertical coordinates of the second vertex of the first target triangle, respectively, C ,y C are the horizontal and vertical coordinates of the third vertex of the first target triangle, and u and v are weights.

[0018] In an embodiment of the present application, determining a second pixel point corresponding to the first pixel point in the second image according to the weight and the pixel coordinates of the three vertices of the second target triangle includes: determining the second pixel point according to formula (2):

[0019]

[0020] Among them, x p’ ,y p’ are the horizontal and vertical coordinates of the second pixel, respectively. A’ ,y A’ are the horizontal and vertical coordinates of the first vertex of the second target triangle, respectively. B’ ,y B’ are the horizontal and vertical coordinates of the second vertex of the second target triangle, respectively. C’ ,y C’ are the horizontal and vertical coordinates of the third vertex of the second target triangle, and u and v are weights.

[0021] In an embodiment of the present application, the image processing method further includes:

[0022] In a case where the color values of all the pixels in the first area are correspondingly assigned to all the second pixels in the second area, the first area in the first image is deleted.

[0023] A second aspect of the present application provides an anti-collision control method applied to an aerial work platform. The aerial work platform includes a work platform and an anti-collision control system. The anti-collision control system includes a radar sensor for detecting obstacles around the work platform and a camera for capturing images of the environment around the work platform. The anti-collision control method includes:

[0024] Get the environment image captured by the camera;

[0025] Input the environment image into the obstacle recognition model to obtain the obstacle recognition result;

[0026] Determine whether an obstacle is identified in the target direction according to the obstacle recognition result;

[0027] When an obstacle is identified, obstacle information of the target obstacle closest to the work platform is determined based on the detection signal obtained from the radar sensor;

[0028] Execute anti-collision strategies based on obstacle information.

[0029] In the embodiment of the present application, the obstacle information includes the minimum distance and relative speed between the target obstacle and the working platform.

[0030] In an embodiment of the present application, executing an anti-collision strategy based on obstacle information includes:

[0031] Determine the time of collision based on the minimum distance and relative speed;

[0032] Determine the risk level based on the time of collision;

[0033] Execute the corresponding anti-collision strategy according to the determined risk level.

[0034] In the embodiment of the present application, determining the risk level according to the collision occurrence time includes at least one of the following:

[0035] If the collision occurs within the first interval, the risk level is determined to be level one, wherein the anti-collision strategy corresponding to level one includes only issuing an alarm prompt;

[0036] If the collision occurs within a second interval that is smaller than the first interval, the risk level is determined to be a level 2 risk level, wherein the anti-collision strategy corresponding to the level 2 risk level includes issuing an alarm and controlling the aerial work platform to decelerate.

[0037] If the collision occurs within a third interval that is smaller than the second interval, the risk level is determined to be a level three risk level, wherein the anti-collision strategy corresponding to the level three risk level includes issuing an alarm and controlling the aerial work platform to move at a speed after deceleration;

[0038] If the collision occurs within a fourth interval that is smaller than the third interval, the risk level is determined to be a fourth risk level, where the anti-collision strategy corresponding to the fourth risk level includes sounding an alarm and controlling the aerial work platform to perform emergency braking.

[0039] In the embodiment of the present application, the target direction is the movement direction of the working platform.

[0040] In an embodiment of the present application, the camera includes a side camera for capturing a side environmental image of the work platform and a top camera for capturing an environmental image of the work platform from a top-down angle, and the side camera and the top camera have a capturing overlapping area. The anti-collision control method further includes:

[0041] The first image taken by the side camera and the second image taken by the top camera are processed using the above-mentioned image processing method.

[0042] A third aspect of the present application provides a processor configured to execute the above-mentioned image processing method.

[0043] A fourth aspect of the present application provides a processor configured to execute the above-mentioned anti-collision control method.

[0044] A fifth aspect of the present application provides an anti-collision control system, which is applied to an aerial work platform. The aerial work platform includes a work platform. The anti-collision control system includes:

[0045] Radar sensors for detecting obstacles around the work platform;

[0046] A camera for capturing images of the environment surrounding the work platform; and

[0047] The processor mentioned above.

[0048] A sixth aspect of the present application provides an aerial work platform, comprising:

[0049] work platforms; and

[0050] The anti-collision control system mentioned above.

[0051] In a seventh aspect, the present application provides a machine-readable storage medium having instructions stored thereon. When the instructions are executed by a processor, the processor implements the above-mentioned image processing method.

[0052] In an eighth aspect, the present application provides a machine-readable storage medium having instructions stored thereon. When the instructions are executed by a processor, the processor implements the above-mentioned anti-collision control method.

[0053] Through the above technical solution, when there is an overlapping area between the side camera and the top camera in multi-screen monitoring, the image of the overlapping area in the side camera image is mapped to the image of the top camera, so as to merge the images of the side camera and the top camera and provide a more optimized visual interface for the operator.

[0054] In addition, the anti-collision control combines the position information and speed information of the detected obstacles to formulate anti-collision strategies, optimize the execution of anti-collision actions, and improve user comfort.

[0055] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present application but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0057] Figure 1A and Figure 1BThe radar arrangement of the anti-collision control system in the prior art is shown;

[0058] Figure 2 The structure diagram of the radar bracket in the anti-collision control system for an aerial work platform according to an embodiment of the present application is schematically shown;

[0059] Figure 3A and Figure 3B The following schematically illustrates the arrangement of a radar bracket in an anti-collision control system for an aerial work platform according to an embodiment of the present application;

[0060] Figure 4A and Figure 4B The schematic diagram shows the radar detection coverage of the anti-collision control system for aerial work platforms according to an embodiment of the present application, wherein Figure 4A is an axonometric view. Figure 4B It is a top view;

[0061] Figure 5A and Figure 5B The following schematically illustrates the detection areas of the radar of the anti-collision control system for aerial work platforms according to the embodiments of the present application: Figure 5A The figure shows the detection area without considering a certain range in front of the working platform. Figure 5B The figure shows the detection area when a certain range in front of the working platform is considered;

[0062] Figure 6 Schematically illustrates a display interface for visual monitoring of an anti-collision control system for an aerial work platform according to an embodiment of the present application;

[0063] Figure 7 Schematically shows a flow chart of an image processing method for an aerial work platform according to an embodiment of the present application;

[0064] Figure 8 Schematically shows Figure 7 The triangle mapping relationship used in the image processing method;

[0065] Figure 9 A flowchart of an anti-collision control method for an aerial work platform according to an embodiment of the present application is schematically shown.

[0066] Description of Reference Numerals

[0067] 101 Left radar module 102 Right radar module

[0068] 103 rear radar module 104 support rod

[0069] 105 top detection radar 200 radar bracket

[0070] 201 mounting surface 202 base

[0071] 2021 base plate 2022 mounting plate

[0072] 2023 limit mechanism 300 mm wave radar

[0073] 400 side camera DETAILED DESCRIPTION

[0074] The following describes the specific implementation of the embodiment of the present application in detail with reference to the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiment of the present application and is not intended to limit the embodiment of the present application.

[0075] It should be noted that if the implementation methods of this application involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0076] If there are descriptions involving "first", "second", etc. in the embodiments of this application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features specified as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0077] The "left side", "right side" and "rear side" of the work platform involved in the implementation mode of the present application are defined relative to the "front side" of the work platform. The "front side" of the work platform refers to the side where the operating table is provided, that is, the side that the staff faces when operating the operating table.

[0078] Figure 2 The structure diagram of the radar bracket 200 in the anti-collision control system for the aerial work platform according to an embodiment of the present application is schematically shown. Figure 3A and Figure 3B The schematic diagram shows the arrangement of the radar bracket 200 in the anti-collision control system for aerial work platforms according to an embodiment of the present application. Figure 2 、 Figure 3A and Figure 3BIn an embodiment of the present application, a collision avoidance control system for an aerial work platform is provided. The aerial work platform may include a work platform and other components. Depending on the type of the aerial work platform, the other components may include, but are not limited to, a main body (e.g., a walking mechanism), an arm, a lifting mechanism, etc. The collision avoidance control system may include:

[0079] A left radar module 101, a right radar module 102, and a rear radar module 103 are respectively arranged on the left, right, and rear sides of the work platform, wherein each radar module includes a radar bracket 200 and multiple millimeter-wave radars 300. The radar bracket 200 includes multiple mounting surfaces 201. The multiple mounting surfaces 201 have different angles relative to the reference surface. The multiple millimeter-wave radars 300 are mounted on the multiple mounting surfaces 201.

[0080] Support rod 104, mounted on the work platform; and

[0081] The top detection radar 105 is installed on the top of the support rod 104 and is used to detect the space above the work platform.

[0082] Specifically, in one example, the left radar module 101 can be mounted on the left bottom of the work platform, the right radar module 102 can be mounted on the right bottom of the work platform, and the rear radar module 103 can be mounted on the rear bottom of the work platform. Radar bracket 200 can include a base 202 and multiple mounting surfaces 201 fixed to the base 202. Base 202 can include a bottom plate 2021 and multiple mounting plates 2022 extending from the bottom plate 2021. Multiple mounting surfaces 201 can be mounted on the mounting plates 2022.

[0083] In one example, the mounting surface 201 can be pin-connected to the mounting plate 2022 and have a limiting mechanism 2023, so that the mounting surface 201 can rotate within a certain angle range relative to the mounting plate 2022 and can be maintained at a set angle (for example, by adding a damping or fixing device). The base plate 2021 can be used to be fixed to the work platform (for example, by a threaded connection or other fixed connection method). Each mounting surface 201 can be provided with a fixing mechanism (for example, a mounting hole) or a clamping mechanism for fixing the radar detection working surface of the millimeter wave radar 300 to the mounting surface 201.

[0084] In one example, multiple mounting surfaces 201 can be arranged along the length of the radar module (e.g., along the bottom plate 2021), with the detection working surface of the millimeter-wave radar 300 parallel to the corresponding mounting surface 201. For example, the number of mounting surfaces 201 to be used and the angle of each mounting surface 201 relative to a reference surface can be set based on the detection range (detection pitch angle) and number of millimeter-wave radars 300 to be used. The reference surface here can include, but is not limited to, the bottom plate plane, the horizontal surface after the radar module is installed on the work platform, and one of the multiple mounting surfaces 201 to be used.

[0085] In one example, the number of millimeter-wave radars 300 used can be less than the number of mounting surfaces 201. For example, the number of mounting surfaces 201 of the radar bracket 200 can be 4, and the number of millimeter-wave radars 300 used can be 3. The number of millimeter-wave radars 300 can be determined based on the detection range (detection pitch angle) and detection coverage requirements of the millimeter-wave radar 300. For example, if the detection range of the millimeter-wave radar 300 is ±15°, the angles of the three mounting surfaces 201 can be set to differ by 30°, and then the three millimeter-wave radars 300 are installed on these three mounting surfaces 201. As a result, the working detection surfaces of the three millimeter-wave radars 300 also differ by 30° in sequence. In this way, the three millimeter-wave radars 300 can cover a detection range of 90°. Using the radar arrangement method of the embodiment of the present application, 90° detection coverage can basically be achieved for the left, right and rear sides of the work platform.

[0086] In addition, in this embodiment of the present application, a top-detection radar 105 mounted on the support rod 104 detects the area above the work platform, further eliminating detection blind spots. Examples of top-detection radar 105 include, but are not limited to, ultrasonic radar, laser radar, and millimeter-wave radar. Preferably, top-detection radar 105 can be a millimeter-wave radar, similar to the side radar.

[0087] In an embodiment of the present application, the support rod 104 can be mounted on a guardrail of the work platform. For example, the support rod 104 can be located on the side of the work platform near the front. In one example, the support rod can be retractable, for example, an electrically retractable rod (e.g., a lead screw structure).

[0088] Figure 4A and Figure 4B The schematic diagram shows the radar detection coverage of the anti-collision control system for aerial work platforms according to an embodiment of the present application. The radar arrangement of the embodiment of the present application can solve the problem of the small detection range and large blind spot of the current anti-collision control system for aerial work platforms.

[0089] Figure 5A and Figure 5BThe following schematically illustrates the detection area of a radar in an example of an anti-collision control system for an aerial work platform according to an embodiment of the present application: Figure 5A The figure shows the detection area without considering a certain range in front of the working platform. Figure 5B The figure shows the detection area when considering a certain range in front of the work platform. In this example, when the target range in front of the work platform (e.g., 3 meters) is not considered, the detection coverage rate for the target ranges in the four directions (e.g., 3 meters) upward, left, right, and rearward is 92%. When the target range in front of the work platform (e.g., 3 meters) is considered, the detection coverage rate for the target ranges in the five directions (e.g., 3 meters) upward, left, right, front, and rear is 80.4%.

[0090] In a further embodiment of the present application, the anti-collision control system may further include a visual system. Specifically, the anti-collision control system may further include:

[0091] A side camera 400 is used to capture side environmental images of the work platform; and

[0092] The top camera (not shown in the figure) is installed on the support rod and is used to capture the environmental image of the working platform from a bird's-eye view.

[0093] Specifically, the side camera 400 may include:

[0094] The left camera is mounted on the radar bracket 200 of the left radar module 101 and is used to capture images of the environment on the left side of the work platform;

[0095] The right camera is mounted on the radar bracket 200 of the right radar module 102 and is used to capture images of the environment on the right side of the work platform;

[0096] The rear camera is mounted on the radar bracket 200 of the rear radar module 103 and is used to capture images of the environment at the rear of the work platform.

[0097] Specifically, any one of the left camera, the right camera, and the rear camera has a photographing overlapping area with the top camera.

[0098] The rear camera has a shooting overlap area with the left camera or the right camera.

[0099] In the embodiment of the present application, the side camera 400 can be set on the radar bracket 200. For example, the left side camera

[0100] By fusing the images captured by the top camera and the side cameras 400, a bird's-eye view of the work platform's surroundings can be provided. Therefore, the collision avoidance control system may also include a display for displaying the images captured by the cameras. The display may be mounted, for example, on the operating console of the work platform. Figure 6An example of an image displayed on a display is shown. Figure 6 As shown, the four boxes at the bottom of the figure display the video images collected by the left, rear, right and top cameras in sequence. Figure 6 The upper area in the figure shows the working environment around the working platform.

[0101] By adding a visual system, the embodiment of the present application can provide workers with a visual interface around the work platform, which helps to improve work efficiency and safety.

[0102] In the embodiment of the present application, the anti-collision control system may further include a processor. The processor may obtain the image captured by the camera, identify the category of the target obstacle and the distance from the work platform according to the image, and may display the category and distance (for example, in Figure 6 (displayed in the upper area of ).

[0103] In addition, the processor can also perform anti-collision measures based on the obstacle signals detected by the millimeter wave radar. These and other functions that the processor can perform will be further described below.

[0104] In visual surveillance using multiple perspectives (multiple cameras), since the top camera shoots downward from the work platform, its shooting range will overlap with the shooting range of the side camera 400. If the target obstacle is in this overlapping area, the same obstacle will appear in the monitoring image of the top camera and the monitoring image of the side camera 400, and appear in different positions on the monitoring images, which can easily cause confusion for workers. Therefore, the obstacle information obtained by different cameras can be fused. In view of this, one embodiment of the present application provides an image processing method that can be applied to an anti-collision control system or aerial work platform that includes a camera.

[0105] Specifically, Figure 7 The flowchart of the image processing method for an aerial work platform according to an embodiment of the present application is schematically shown. The anti-collision control system may include or the work platform may be provided with a side camera for capturing side environmental images of the work platform and a top camera for capturing environmental images of the work platform from a top-down angle, wherein the side camera and the top camera have overlapping shooting areas, such as Figure 7 As shown, the image processing method may include the following steps.

[0106] In step S710, a plurality of first points of interest in a first area corresponding to the overlapping shooting area in the first image taken by the side camera are determined, and a plurality of second points of interest corresponding to the plurality of first points of interest in a second area corresponding to the overlapping shooting area in the second image taken by the top camera are determined, wherein the plurality of first points of interest form a first mesh consisting of a plurality of first triangles with the first points of interest as vertices, and the plurality of second points of interest form a second mesh consisting of a plurality of second triangles with the second points of interest as vertices.

[0107] Associating pixels in the first area with pixels in the second area based on a correspondence between a first triangle in the first grid and a second triangle in the second grid; and

[0108] Assign the color values of the pixels in the first area to the corresponding pixels in the second area.

[0109] Specifically, information from the top and side cameras can be fused based on triangular interpolation mapping. In this step, a correspondence is obtained between all pixels in the overlapping region of the images captured by the side cameras and all pixels in the overlapping region of the images captured by the top cameras. However, the number of points of interest that can be pre-selected is limited. Therefore, embodiments of the present invention can use triangles to divide the surveillance scene. First, a finite number of point pairs can be calibrated for the overlapping region in the images captured by the side and top cameras, and then uniformly triangulated to form a grid composed of triangles that covers the overlapping region. The subsequent mapping process can use a one-by-one mapping strategy, mapping each point within the triangular region to the image captured by the top camera (e.g., a bird's-eye view of the scene). The point pair calibration method can, for example, include placing a calibration object (point) at a certain location in the overlapping region. The side and top cameras capture images (e.g., a side image and a top image, respectively). The pixel coordinates of the center of the calibration object in the side and top images can be determined, and these two pixel coordinates can be associated. Then the position of the calibration object is changed, and the side camera and the top camera take images again respectively. The pixel coordinates of the center of the calibration object after the change of position are determined again and associated. After changing the position of the calibration object multiple times, a set of interest points for the side image and a corresponding set of interest points for the top image can be obtained. According to these interest point sets, a grid composed of triangles covering the overlapping area of the side image and a grid composed of triangles covering the overlapping area of the top image can be formed respectively, and the triangles in the two grids can correspond one to one.

[0110] In an embodiment of the present application, according to the correspondence between the first triangle in the first grid and the second triangle in the second grid, associating the pixel points in the first area with the pixel points in the second area may specifically include the following steps.

[0111] In step S720, for any first pixel point other than the first point of interest in the first area, a first target triangle where the first pixel point is located is determined according to the pixel coordinates of the first pixel point.

[0112] Specifically, since the pixel coordinates of the point of interest are known, for any pixel other than the point of interest, as long as the coordinates of the pixel are known, we can know which triangle in the grid it is located in. In other words, the coordinates of the arbitrary pixel and the three vertices of the triangle in which it is located are all known.

[0113] In step S730 , a weight is determined according to the pixel coordinates of the first pixel point and the pixel coordinates of the three vertices of the first target triangle.

[0114] In step S740 , a second target triangle corresponding to the first target triangle in the second image is determined.

[0115] In step S750 , a second pixel point corresponding to the first pixel point in the second image is determined according to the weight and the pixel coordinates of the three vertices of the second target triangle.

[0116] Specifically, each triangle mapping relationship can be calculated using a triangle interpolation formula. Figure 8 Schematically shows Figure 7 The triangle mapping relationship used in the image processing method. Figure 8 As shown,

[0117] Assigning the color value of a pixel in the first area to a corresponding pixel in the second area may include step S760: assigning the color value of the first pixel to the second pixel; and may also include assigning the color value of the first point of interest to the corresponding second point of interest.

[0118] Specifically, triangle ABC is mapped onto triangle A'B'C'. Point P in triangle ABC is mapped to point P'. Given that any point in a triangle can be represented as the weighted sum of the three vertices ABC, after point P is mapped to P', point P' can still be represented as the weighted sum of the three vertices A'B'C' of the new triangle, with the same weights. Therefore, all we need to do is calculate these three weights.

[0119] Specifically, determining the weights according to the pixel coordinates of the first pixel point and the pixel coordinates of the three vertices of the first target triangle includes: determining the weights according to formula (1):

[0120]

[0121] Among them, x p ,yp are the horizontal and vertical coordinates of the first pixel, respectively, A ,y A are the horizontal and vertical coordinates of the first vertex of the first target triangle, respectively, B ,y B are the horizontal and vertical coordinates of the second vertex of the first target triangle, respectively, C ,y C are the horizontal and vertical coordinates of the third vertex of the first target triangle, u and v are weights, each weight can be a positive number less than 1, and satisfies u+v<1.

[0122] According to the two equations in formula (1), u and v can be obtained by solving them.

[0123] After the first target triangle is determined, a second target triangle corresponding to the first target triangle can be determined, that is, the coordinates of the three vertices of the second target triangle are known.

[0124] Then substitute the obtained u and v into the following formula (2) to obtain the second pixel point (coordinates) corresponding to the first pixel point:

[0125]

[0126] Among them, x p’ ,y p’ are the horizontal and vertical coordinates of the second pixel, respectively. A’ ,y A’ are the horizontal and vertical coordinates of the first vertex of the second target triangle, respectively. B’ ,y B’ are the horizontal and vertical coordinates of the second vertex of the second target triangle, respectively. C’ ,y C’ are the abscissa and ordinate of the third vertex of the second target triangle respectively.

[0127] After determining the pixel (coordinates) in the top image corresponding to the arbitrary pixel (coordinates) in the side image, the color value (e.g., RGB value) of the arbitrary pixel can be assigned to the corresponding pixel in the top image. The same mapping operation can be performed on all pixels covered by the triangular mesh in the side image to complete the mapping of the entire overlapping area.

[0128] Although the above description is about determining the corresponding pixel point in the top image from any pixel point in the side image, those skilled in the art will understand that it is also possible to reversely infer the corresponding pixel point in the side image based on any pixel point in the top image, and then obtain the color value of the corresponding pixel point and assign it to the arbitrary pixel point in the top image.

[0129] Optionally, after the color values of all pixels in the first area are assigned to all second pixels in the second area, the first area in the first image can be deleted. For example, the overlapping area can be cropped out of the side image and not displayed in the monitoring screen.

[0130] In an alternative or additional embodiment of the present application, the field of view of two adjacent side cameras may also have overlapping shooting areas. In this case, in order to fuse the environmental information of the overlapping areas, the Laplacian pyramids of the image data of the two side cameras can be calculated separately, and then the mask matrix is set, and then the mask Gaussian pyramid is calculated, and finally Laplacian reconstruction is performed to obtain the final fusion result.

[0131] In addition, the current anti-collision control system or method only utilizes distance information and lacks the use of obstacle speed information, resulting in unreasonable anti-collision control actions and poor comfort of the anti-collision actions, which affects the user experience of the aerial work platform anti-collision control system.

[0132] In view of this, an embodiment of the present application also provides an anti-collision control method. Figure 9 The flowchart of the anti-collision control method for an aerial work platform according to an embodiment of the present application is schematically shown. The anti-collision control method can be applied to an aerial work platform, which includes a work platform and an anti-collision control system, wherein the anti-collision control system includes a radar sensor for detecting obstacles around the work platform and a camera for capturing images of the environment around the work platform. For example, the anti-collision control system can be the anti-collision control system of the above embodiment. Figure 9 As shown, in an embodiment of the present application, the anti-collision control method may include the following steps.

[0133] In step S910 , an environment image captured by a camera is acquired.

[0134] In step S920 , the environment image is input into the obstacle recognition model to obtain an obstacle recognition result.

[0135] Specifically, for example, after the processor acquires the environment image taken by the camera, the obstacle recognition model can be used to identify obstacles in the environment image. In one example, the obstacle recognition model can be a target detection model based on deep learning. The target obstacle database (training set) used to train the target detection model can be collected offline, and the target detection model can also be trained offline in a pre-running mode. The method for training the target detection model is known to those skilled in the art and will not be described in detail here. If the target detection model identifies an obstacle, the identified obstacle can be marked with a detection frame (bounding box), and the type and location information of the identified obstacle can be obtained.

[0136] In step S930 , it is determined whether an obstacle is recognized in the target direction according to the obstacle recognition result.

[0137] Specifically, in one embodiment, the target direction may be a direction determined based on the on-site environment. In some embodiments, the camera captures images of the environment surrounding the work platform, but for collision avoidance, what may be of concern is whether there are obstacles in front of the direction of movement of the work platform. Therefore, the obstacle recognition model only needs to identify obstacles in the target direction. For example, the processor can obtain the direction of movement of the work platform and determine the direction in front of its direction of movement as the target direction. In this case, the processor only needs to obtain the image captured by the camera in front of the direction of movement, recognize the image, and output the recognition result. The recognition result may include the type and location information of the obstacle. In addition, the processor can discard images captured by other cameras.

[0138] In step S940 , when an obstacle is identified, obstacle information of a target obstacle closest to the working platform is determined based on the detection signal obtained from the radar sensor.

[0139] Specifically, if the obstacle recognition model identifies a target obstacle, the processor can obtain the detection signal obtained by the radar sensor and determine obstacle information for the obstacle in the target direction based on the detection signal. If there are multiple obstacles in the target direction, the processor can determine obstacle information for the target obstacle closest to the work platform based on the detection signal, including its position information (minimum distance) and relative speed information relative to the work platform.

[0140] In step S950 , an anti-collision strategy is executed according to the obstacle information.

[0141] Specifically, the collision occurrence time T can be calculated based on the position information (minimum distance d) and relative speed information (relative linear speed v) of the target obstacle:

[0142]

[0143] If T falls below a certain threshold, the anti-collision control system can alert the operator, such as with a photoelectric alarm. If the operator fails to respond effectively and in a timely manner, the system can proactively intervene with deceleration control or movement restriction to prevent an accident.

[0144] Specifically, after determining the collision time T, an anti-collision strategy can be formulated based on the collision time T. In general, the risk level is determined based on the collision time, and the corresponding anti-collision strategy is executed based on the determined risk level. For example, the processor can activate an alarm device (such as an audible / visual alarm device), control the speed / direction of the aerial work platform, control the speed / direction of the work platform, and implement emergency braking.

[0145] Determining the risk level based on the time of collision includes at least one of the following:

[0146] If the collision occurs within the first interval, the risk level is determined to be level one, wherein the anti-collision strategy corresponding to level one includes only issuing an alarm prompt;

[0147] If the collision occurs within a second interval that is smaller than the first interval, the risk level is determined to be a level 2 risk level, wherein the anti-collision strategy corresponding to the level 2 risk level includes issuing an alarm and controlling the aerial work platform to decelerate.

[0148] If the collision occurs within a third interval that is smaller than the second interval, the risk level is determined to be a level three risk level, wherein the anti-collision strategy corresponding to the level three risk level includes issuing an alarm and controlling the aerial work platform to move at a speed after deceleration;

[0149] If the collision occurs within a fourth interval that is smaller than the third interval, the risk level is determined to be a fourth risk level, where the anti-collision strategy corresponding to the fourth risk level includes sounding an alarm and controlling the aerial work platform to perform emergency braking.

[0150] For example, in one example, collision avoidance measures (or actions) may include:

[0151] (1) When 2s≤T<3s, the risk level is 1, and an alarm is issued;

[0152] (2) When 1.5s≤T<2s, the risk level is 2, an alarm is issued, and deceleration control is performed;

[0153] (3) When 1s≤T≤1.5s, the risk level is 3, an alarm is issued, and the platform moves at a snail's pace (for example, the control current of the hydraulic cylinder driving the working platform is less than 20% of the maximum current);

[0154] (4) When T < 1s, the risk level is 4, an alarm is issued, and emergency braking is performed.

[0155] In an embodiment of the present application, the anti-collision control method may further include processing the images taken by the side camera and the images taken by the top camera using the image processing method of any of the above embodiments.

[0156] The above-mentioned scheme of the embodiment of the present application formulates an anti-collision control strategy by calculating the collision occurrence time T, which can fully utilize the relative speed relationship between the working platform and the obstacle, and the anti-collision action is more comfortable.

[0157] An embodiment of the present application provides a processor configured to execute the image processing method of any of the above embodiments.

[0158] Specifically, the processor may be configured to:

[0159] Determine a plurality of first interest points in a first area corresponding to the overlapping area in a first image captured by the side camera, and determine a plurality of second interest points corresponding to the plurality of first interest points in a second image captured by the top camera, wherein the plurality of first interest points form a first mesh consisting of a plurality of first triangles with the first interest points as vertices, and the plurality of second interest points form a second mesh consisting of a plurality of second triangles with the second interest points as vertices;

[0160] For any first pixel point other than the first point of interest in the first area, determining a first target triangle where the first pixel point is located according to the pixel coordinates of the first pixel point;

[0161] Determine a weight according to the pixel coordinates of the first pixel point and the pixel coordinates of the three vertices of the first target triangle;

[0162] determining a second target triangle in the second image corresponding to the first target triangle;

[0163] Determine a second pixel point corresponding to the first pixel point in the second image according to the weight and the pixel coordinates of the three vertices of the second target triangle;

[0164] Assign the color value of the first pixel to the second pixel.

[0165] In the embodiment of the present application, determining the weight according to the pixel coordinates of the first pixel point and the pixel coordinates of the three vertices of the first target triangle includes: determining the weight according to formula (1):

[0166]

[0167] Among them, x p ,y p are the horizontal and vertical coordinates of the first pixel, respectively, A ,y A are the horizontal and vertical coordinates of the first vertex of the first target triangle, respectively, B ,y B are the horizontal and vertical coordinates of the second vertex of the first target triangle, respectively, C ,yC are the horizontal and vertical coordinates of the third vertex of the first target triangle, and u and v are weights.

[0168] In an embodiment of the present application, determining a second pixel point corresponding to the first pixel point in the second image according to the weight and the pixel coordinates of the three vertices of the second target triangle includes: determining the second pixel point according to formula (2):

[0169]

[0170] Among them, x p’ ,y p’ are the horizontal and vertical coordinates of the second pixel, respectively. A’ ,y A’ are the horizontal and vertical coordinates of the first vertex of the second target triangle, respectively. B’ ,y B’ are the horizontal and vertical coordinates of the second vertex of the second target triangle, respectively. C’ ,y C’ are the horizontal and vertical coordinates of the third vertex of the second target triangle, and u and v are weights.

[0171] In this embodiment of the present application, the processor is further configured to:

[0172] In a case where the color values of all the pixels in the first area are correspondingly assigned to all the second pixels in the second area, the first area in the first image is deleted.

[0173] An embodiment of the present application provides a processor configured to execute the anti-collision control method of any of the above embodiments.

[0174] Specifically, the processor may be configured to:

[0175] Get the environment image captured by the camera;

[0176] Input the environment image into the obstacle recognition model to obtain the obstacle recognition result;

[0177] Determine whether an obstacle is identified in the target direction according to the obstacle recognition result;

[0178] When an obstacle is identified, obstacle information of the target obstacle closest to the work platform is determined based on the detection signal obtained from the radar sensor;

[0179] Execute anti-collision strategies based on obstacle information.

[0180] In the embodiment of the present application, the obstacle information includes the minimum distance and relative speed between the target obstacle and the working platform.

[0181] In an embodiment of the present application, executing an anti-collision strategy based on obstacle information includes:

[0182] Determine the time of collision based on the minimum distance and relative speed;

[0183] Determine the risk level based on the time of collision;

[0184] Execute the corresponding anti-collision strategy according to the determined risk level.

[0185] In the embodiment of the present application, determining the risk level according to the collision occurrence time includes at least one of the following:

[0186] If the collision occurs within the first interval, the risk level is determined to be level one, wherein the anti-collision strategy corresponding to level one includes only issuing an alarm prompt;

[0187] If the collision occurs within a second interval that is smaller than the first interval, the risk level is determined to be a level 2 risk level, wherein the anti-collision strategy corresponding to the level 2 risk level includes issuing an alarm and controlling the aerial work platform to decelerate.

[0188] If the collision occurs within a third interval that is smaller than the second interval, the risk level is determined to be a level three risk level, wherein the anti-collision strategy corresponding to the level three risk level includes issuing an alarm and controlling the aerial work platform to move at a speed after deceleration;

[0189] If the collision occurs within a fourth interval that is smaller than the third interval, the risk level is determined to be a fourth risk level, where the anti-collision strategy corresponding to the fourth risk level includes sounding an alarm and controlling the aerial work platform to perform emergency braking.

[0190] In the embodiment of the present application, the target direction is the movement direction of the working platform.

[0191] In this embodiment of the present application, the processor is further configured to:

[0192] The first image taken by the side camera and the second image taken by the top camera are processed using the above-mentioned image processing method.

[0193] The present application provides an anti-collision control system for an aerial work platform. The aerial work platform includes a work platform. The anti-collision control system includes:

[0194] Radar sensors for detecting obstacles around the work platform;

[0195] A camera for capturing images of the environment surrounding the work platform; and

[0196] The processor is configured to execute the anti-collision control method of any of the above embodiments.

[0197] In an embodiment of the present application, the anti-collision control system may be the anti-collision control system in any of the above embodiments, and accordingly, the radar sensor may be various radar modules (for example, side radar modules and top radar modules) in any of the above embodiments.

[0198] Examples of processors may include, but are not limited to, single chip microcomputers, microprocessors, field programmable gate arrays (FPGAs), programmable logic controllers (PLCs), digital signal processors (DSPs), application specific integrated circuits (ASICs), state machines, etc.

[0199] The present application provides an aerial work platform, comprising:

[0200] work platforms; and

[0201] The anti-collision control system of any of the above embodiments.

[0202] An embodiment of the present application provides a machine-readable storage medium having instructions stored thereon. When the instructions are executed by a processor, the processor implements the image processing method of any of the above embodiments.

[0203] An embodiment of the present application provides a machine-readable storage medium having instructions stored thereon. When the instructions are executed by a processor, the processor implements the anti-collision control method of any of the above embodiments.

[0204] The technical solutions provided in the above embodiments of the present application may have at least one of the following beneficial effects:

[0205] (1) The anti-collision control (monitoring) system of the embodiment of the present application has a large detection range, with a coverage rate of more than 80% of the space within 3 meters above the working platform, a small blind spot, and greater safety.

[0206] (2) The anti-collision control (monitoring) system of the embodiment of the present application can provide a visual interface of the working environment around the work platform, making it convenient for workers inside the work platform to observe the surrounding environment.

[0207] (3) The anti-collision control (monitoring) system of the embodiment of the present application mainly adopts millimeter wave radar, which can ensure the safety of the aerial work platform even when it is traveling at high speed (>6km / h).

[0208] (4) The detection results of the anti-collision control (monitoring) system of the embodiment of the present application are not directly used for the execution of anti-collision control actions. Instead, the detection results are comprehensively judged through the image data collected by the camera, so that the detection results are more reliable and the false alarm rate of the anti-collision control system can be effectively reduced.

[0209] (5) The embodiment of the present application proposes formulating an anti-collision control strategy by calculating the collision occurrence time T, which can fully utilize the relative speed relationship between the working platform and the obstacle, and the anti-collision action is more comfortable.

[0210] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0211] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: Applied to an aerial work platform, the aerial work platform includes a work platform, the work platform is provided with a side camera for capturing a side environmental image of the work platform and a top camera for capturing an environmental image of the work platform from a top-down angle, the side camera and the top camera having an overlapping shooting area, and the image processing method includes: Determine a plurality of first points of interest in a first area corresponding to the overlapping shooting area in a first image captured by the side camera, and determine a plurality of second points of interest corresponding to the plurality of first points of interest in a second area corresponding to the overlapping shooting area in a second image captured by the top camera, wherein the plurality of first points of interest form a first mesh consisting of a plurality of first triangles with the first points of interest as vertices, and the plurality of second points of interest form a second mesh consisting of a plurality of second triangles with the second points of interest as vertices; Associating pixels in the first area with pixels in the second area based on a correspondence between a first triangle in the first grid and a second triangle in the second grid; Assigning color values of pixels in the first area to corresponding pixels in the second area; In a case where the color values of all pixels in the first area are correspondingly assigned to all pixels in the second area, the first area in the first image is deleted.

2. The image processing method according to claim 1, wherein: The associating the pixel points in the first area with the pixel points in the second area according to the correspondence between the first triangle in the first grid and the second triangle in the second grid includes: For any first pixel point other than the first point of interest in the first area, determining a first target triangle where the first pixel point is located according to the pixel coordinates of the first pixel point; Determine a weight according to the pixel coordinates of the first pixel point and the pixel coordinates of the three vertices of the first target triangle; determining a second target triangle in the second image corresponding to the first target triangle; Determine a second pixel point corresponding to the first pixel point in the second image according to the weight and the pixel coordinates of the three vertices of the second target triangle; Assigning the color value of the pixel point in the first area to the corresponding pixel point in the second area includes: assigning the color value of the first pixel point to the second pixel point.

3. The image processing method according to claim 2, wherein: The determining of the weights according to the pixel coordinates of the first pixel point and the pixel coordinates of the three vertices of the first target triangle includes: determining the weights according to formula (1): Formula (1) in, x p , y p are the horizontal and vertical coordinates of the first pixel point respectively, x A , y A are the abscissa and ordinate of the first vertex of the first target triangle, x B , y B are the abscissa and ordinate of the second vertex of the first target triangle, x C , y C are the abscissa and ordinate of the third vertex of the first target triangle, u , v is the weight.

4. The image processing method according to claim 3, wherein: Determining a second pixel point corresponding to the first pixel point in the second image according to the weight and the pixel coordinates of the three vertices of the second target triangle includes: determining the second pixel point according to formula (2): Formula (2) in, x p’ , y p’ are the horizontal and vertical coordinates of the second pixel point respectively, x A’ , y A’ are the abscissa and ordinate of the first vertex of the second target triangle, x B’ , y B’ are the abscissa and ordinate of the second vertex of the second target triangle, respectively, x C’ , y C’ are the abscissa and ordinate of the third vertex of the second target triangle, u , v is the weight.

5. A collision prevention control method, characterized in that: Applied to an aerial work platform, the aerial work platform includes a work platform and an anti-collision control system, the anti-collision control system includes a radar sensor for detecting obstacles around the work platform and a camera for capturing images of the environment around the work platform. The anti-collision control method includes: Acquire the environment image captured by the camera; Inputting the environment image into an obstacle recognition model to obtain an obstacle recognition result; determining whether an obstacle is identified in the target direction according to the obstacle identification result; When the obstacle is identified, determining obstacle information of a target obstacle closest to the work platform based on the detection signal obtained from the radar sensor; executing an anti-collision strategy according to the obstacle information; The camera includes a side camera for capturing a side environmental image of the work platform and a top camera for capturing an environmental image of the work platform at a top-down angle, and the side camera and the top camera have a capturing overlapping area. The anti-collision control method further includes: The first image taken by the side camera and the second image taken by the top camera are processed using the image processing method according to any one of claims 1 to 4.

6. The anti-collision control method according to claim 5, characterized in that: The obstacle information includes a minimum distance and a relative speed between the target obstacle and the working platform.

7. The anti-collision control method according to claim 6, characterized in that: The executing of the anti-collision strategy according to the obstacle information includes: determining a collision occurrence time based on the minimum distance and the relative speed; determining a risk level based on the time of occurrence of said collision; Execute the corresponding anti-collision strategy according to the determined risk level.

8. The anti-collision control method according to claim 7, characterized in that: Determining the risk level according to the collision occurrence time includes at least one of the following: If the collision occurrence time is within the first interval, the risk level is determined to be a level one risk level, wherein the anti-collision strategy corresponding to the level one risk level includes only issuing an alarm prompt; If the collision occurrence time is within a second interval that is smaller than the first interval, the risk level is determined to be a level 2 risk level, wherein the anti-collision strategy corresponding to the level 2 risk level includes issuing an alarm and controlling the aerial work platform to decelerate; If the collision occurrence time is within a third interval that is smaller than the second interval, the risk level is determined to be a level three risk level, wherein the anti-collision strategy corresponding to the level three risk level includes issuing an alarm and controlling the aerial work platform to move at a speed after deceleration; If the collision occurrence time is in a fourth interval smaller than the third interval, the risk level is determined to be a fourth risk level, wherein the anti-collision strategy corresponding to the fourth risk level includes issuing an alarm and controlling the aerial work platform to perform emergency braking.

9. The anti-collision control method according to claim 5, characterized in that: The target direction is the movement direction of the working platform.

10. A processor, characterized in that: The apparatus is configured to execute the image processing method according to any one of claims 1 to 4.

11. A processor, characterized in that: The method is configured to execute the anti-collision control method according to any one of claims 5 to 9.

12. An anti-collision control system, characterized in that: Applied to an aerial work platform, the aerial work platform includes a working platform, and the anti-collision control system includes: a radar sensor for detecting obstacles around the working platform; A camera, configured to capture images of the environment surrounding the work platform; and The processor according to claim 11.

13. An aerial work platform, characterized in that: include: work platform; as well as The anti-collision control system according to claim 12.

14. A machine-readable storage medium, characterized in that The machine-readable storage medium stores instructions, which, when executed by a processor, enable the processor to implement the image processing method according to any one of claims 1 to 4 or the anti-collision control method according to any one of claims 5 to 9.

Images