Milling machine dynamic image enhancement method, device and system and electronic equipment

Abstract

The invention provides a milling machine dynamic image enhancement method, device and system and electronic equipment, and relates to the technical field of engineering machinery. The method comprises the following steps: firstly, acquiring an advancing working face image and a side image of the milling machine; determining a first image of a milling drum according to the operation parameters and the side image of the milling drum on the milling machine; then determining a transparent chassis panorama of the milling machine, a target distance between the milling forbidding target and target acquisition equipment and a second position of the milling forbidding target according to the advancing working face image and the side face image of the milling machine; according to the first position and the target distance, the relative position between the milling forbidding target and the cutting line is determined; and finally, based on the transparent chassis panorama, superposing the first image, the second position and the relative position to obtain a dynamic enhanced image. The precision and efficiency of the milling operation can be improved.

Description

Technical Field

[0001] This application relates to the technical field of construction machinery, and particularly to a method, device, system and electronic device for dynamic image enhancement of a milling machine. Background Art

[0002] A milling machine is a heavy construction machinery used for road construction and maintenance, mainly for removing the surface layer of asphalt or concrete pavement. It cuts the pavement material through a rotating milling drum to achieve the purpose of trimming, renovating or completely removing the pavement.

[0003] When the milling machine is operating, if there are milling - prohibited areas such as manhole covers and bridge expansion joints on the milling operation line, it is necessary for the operator to command the milling machine operator to lift or lower the milling drum at specific positions, so as to avoid milling the milling - prohibited area and prevent damage to the milling machine.

[0004] However, the way of multi - person collaborative operation will result in a large number of milling construction workers, and the position information of manual command may have errors, which affects the efficiency of the operation team. Summary of the Invention

[0005] The embodiments of this application provide a method, device, system and electronic device for dynamic image enhancement of a milling machine, so as to achieve the effect of improving the accuracy and efficiency of milling operations.

[0006] In a first aspect, the embodiments of this application provide a method for dynamic image enhancement of a milling machine, including:

[0007] Obtain the forward working - face image and the side image of the milling machine;

[0008] According to the operation parameters of the milling drum on the milling machine and the side image, determine the first image of the milling drum, where the first image includes the first position of the cutting line of the milling drum in the side image and the projection position of the milling drum;

[0009] According to the forward working - face image and the side image of the milling machine, determine the panoramic view of the transparent chassis of the milling machine, the target distance between the milling - prohibited target and the target acquisition device, and the second position of the milling - prohibited target;

[0010] According to the first position and the target distance, determine the relative position between the milling - prohibited target and the cutting line;

[0011] Based on the panoramic view of the transparent chassis, superimpose the first image, the second position, and the relative position to obtain a dynamically enhanced image.

[0012] In a possible implementation manner, according to the operation parameters of the milling drum on the milling machine and the side image, determining the first image of the milling drum includes:

[0013] Determine the diameter of the milling drum, the milling depth, and the cross-sectional information generated by the milling drum according to the operation parameters;

[0014] Determine the chord length of the cutting line of the cross-section according to the diameter of the milling drum, the milling depth, and the cross-sectional information generated by the milling drum;

[0015] Determine the marked points on the milling drum housing in the side image according to the side image, and the marked points represent the center positions of the milling drums corresponding to different milling depths;

[0016] Map the chord length of the cutting line of the cross-section and the milling drum to the side image according to the marked points to determine the first image of the milling drum.

[0017] In a possible implementation manner, determine the panoramic view of the transparent chassis of the milling machine, the target distance between the milling-prohibited target and the target acquisition device, and the second position of the milling-prohibited target according to the forward working surface image and the side image of the milling machine, including:

[0018] Obtain the panoramic image of the milling machine and the panoramic view of the transparent chassis of the milling machine according to the forward working surface image and the side image of the milling machine;

[0019] Perform target recognition processing on the panoramic image to obtain an initial image, and the initial image includes the bounding box of the milling-prohibited target and the corresponding first category label;

[0020] Perform semantic segmentation processing on the initial image to obtain the second category label corresponding to each pixel in the initial image;

[0021] Determine the second position of the milling-prohibited target according to the bounding box, the first category label, and the second category label;

[0022] Determine the target distance between the milling-prohibited target and the target acquisition device according to the milling depth and the second position.

[0023] In a possible implementation manner, determine the target distance between the milling-prohibited target and the target acquisition device according to the milling depth and the second position, including:

[0024] Determine the calibration parameters of the target acquisition device according to the milling depth, and the calibration parameters include the vertical height and angle of the target acquisition device relative to the ground;

[0025] Convert the image coordinate system to the space coordinate system according to the calibration parameters, and the origin of the space coordinate system is the milling machine;

[0026] Based on the space coordinate system, determine the target distance between the milling-prohibited target and the target acquisition device according to the second position.

[0027] In a possible implementation, the target acquisition device is arranged at the front left position and the front right position of the milling machine, and is used to acquire the forward working face image and the side image of the milling machine.

[0028] In a possible implementation, according to the first position and the target distance, determining the relative position between the milling prohibited target and the cutting line includes:

[0029] Determining the milling speed of the milling machine according to the operation parameters;

[0030] Determining the first dynamic position of the milling prohibited target according to the milling speed and the target distance;

[0031] When the milling prohibited target is not located at the bottom of the milling machine, determining the relative position between the milling prohibited target and the cutting line according to the first position and the first dynamic position.

[0032] In a possible implementation, after determining the first dynamic position of the milling prohibited target according to the milling speed, the method further includes:

[0033] When the milling prohibited target is located at the bottom of the milling machine, determining the predicted position of the milling prohibited target according to the second position;

[0034] Performing a correction process on the first dynamic position according to the predicted position to obtain the second dynamic position of the milling prohibited target;

[0035] Determining the relative position between the milling prohibited target and the cutting line according to the second dynamic position.

[0036] In a second aspect, an embodiment of the present application provides a dynamic image enhancement device for a milling machine, including:

[0037] An acquisition module, configured to acquire the forward working face image and the side image of the milling machine;

[0038] A first determination module, configured to determine a first image of the milling drum according to the operation parameters of the milling drum on the milling machine and the side image, where the first image includes the first position of the cutting line of the milling drum in the side image and the projection position of the milling drum;

[0039] A second determination module, configured to determine the panoramic view of the transparent chassis of the milling machine, the target distance between the milling prohibited target and the target acquisition device, and the second position of the milling prohibited target according to the forward working face image and the side image of the milling machine;

[0040] A third determination module, configured to determine the relative position between the milling prohibited target and the cutting line according to the first position and the target distance;

[0041] A processing module, configured to superimpose the first image, the second position, and the relative position based on the panoramic view of the transparent chassis to obtain a dynamically enhanced image.

[0042] In a possible implementation, the first determination module is specifically configured to:

[0043] Determine the diameter of the milling drum, the milling depth, and the cross-sectional information generated by the milling drum according to the operation parameters;

[0044] Determine the chord length of the cutting line of the cross-section according to the diameter of the milling drum, the milling depth, and the cross-sectional information generated by the milling drum;

[0045] Determine the marked points on the milling drum housing in the side image according to the side image, where the marked points represent the center positions of the milling drums corresponding to different milling depths;

[0046] Map the chord length of the cutting line of the cross-section and the milling drum to the side image according to the marked points, and determine the first image of the milling drum.

[0047] In a possible implementation, the second determination module is specifically configured to:

[0048] Obtain the panoramic image of the milling machine and the panoramic image of the transparent chassis of the milling machine according to the forward working surface image and the side image of the milling machine;

[0049] Perform target recognition processing on the panoramic image to obtain an initial image, where the initial image includes the bounding box of the prohibited milling target and the corresponding first category label;

[0050] Perform semantic segmentation processing on the initial image to obtain the second category label corresponding to each pixel in the initial image;

[0051] Determine the second position of the prohibited milling target according to the bounding box, the first category label, and the second category label;

[0052] Determine the target distance between the prohibited milling target and the target acquisition device according to the milling depth and the second position.

[0053] In a possible implementation, the second determination module is further configured to:

[0054] Determine the calibration parameters of the target acquisition device according to the milling depth, where the calibration parameters include the vertical height and angle of the target acquisition device relative to the ground;

[0055] Convert the image coordinate system to the space coordinate system according to the calibration parameters, where the origin of the space coordinate system is the milling machine;

[0056] Based on the space coordinate system, determine the target distance between the prohibited milling target and the target acquisition device according to the second position.

[0057] In a possible implementation, the target acquisition device is arranged at the front left position and the front right position of the milling machine, and is used to acquire the forward working surface image and the side image of the milling machine.

[0058] In a possible implementation, the third determination module is specifically configured to:

[0059] Determine the milling speed of the milling machine according to the operation parameters;

[0060] Determine the first dynamic position of the milling prohibition target according to the milling speed and the target distance;

[0061] When the milling prohibition target is not located at the bottom of the milling machine, determine the relative position between the milling prohibition target and the cutting line according to the first position and the first dynamic position.

[0062] In a possible implementation, the third determination module is further configured to:

[0063] When the milling prohibition target is located at the bottom of the milling machine, determine the predicted position of the milling prohibition target according to the second position;

[0064] Perform correction processing on the first dynamic position according to the predicted position to obtain the second dynamic position of the milling prohibition target;

[0065] Determine the relative position between the milling prohibition target and the cutting line according to the second dynamic position.

[0066] In a third aspect, an embodiment of the present application provides a milling machine dynamic image enhancement system, including: a frame, a milling drum, a milling drum housing, a target acquisition device, an electronic device, and a video display device;

[0067] The frame is supported on a plurality of traction devices and travels along the working surface, parallel to the traveling axis of the milling machine;

[0068] The milling drum is rotatably supported on the frame for milling the working surface;

[0069] The milling drum housing is supported on the frame for accommodating the milling drum;

[0070] The target acquisition device is used to acquire the forward working surface image and the side image of the milling machine;

[0071] The electronic device is used to execute the above first aspect and / or various possible implementations of the first aspect;

[0072] The video display device is used to display the dynamic enhanced image.

[0073] In a fourth aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor;

[0074] The memory stores computer execution instructions;

[0075] The processor executes the computer-executable instructions stored in the memory, such that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0076] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above first aspect and / or various possible implementations of the first aspect when executed by a processor.

[0077] In a sixth aspect, an embodiment of the present application provides a computer program product including a computer program, which implements the above first aspect and / or various possible implementations of the first aspect when executed by a processor.

[0078] The milling machine dynamic image enhancement method, device, system and electronic device provided by the embodiments of the present application obtain the forward working surface image and the side image of the milling machine, calculate the cutting line position of the milling drum in the side image, and determine the projection position of the milling drum, so as to facilitate the subsequent identification and avoidance of milling-prohibited targets. Furthermore, according to the forward working surface image and the side image, calculate the target distance between the milling-prohibited target and the target acquisition device, and determine the transparent chassis panoramic view of the milling machine and the position of the milling-prohibited target in the image. Thus, the relative position between the milling-prohibited target and the cutting line can be calculated by combining the first position (the cutting line position of the milling drum) and the target distance of the milling-prohibited target, ensuring that the milling machine can accurately avoid the milling-prohibited area. After that, the first image (the cutting line and projection position of the milling drum), the second position of the milling-prohibited target and the relative position are superimposed on the transparent chassis panoramic view. Through the superimposition, the generated dynamic enhanced image can display the working state of the milling machine, the position of the milling drum and the position of the milling-prohibited target in real time. The obtained dynamic enhanced image can help the operator more intuitively understand the working environment, thereby more precisely controlling the operation of the milling machine, avoiding mis-milling of the milling-prohibited area, not only improving the operation efficiency, but also reducing the dependence on manpower and reducing the risk caused by human error. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0080] Figure 1 It is a schematic diagram of the scene when the milling machine operates provided by the embodiment of the present application Figure 1 ;

[0081] Figure 2 It is a schematic diagram of the scene when the milling machine operates provided by the embodiment of the present application Figure 2 ;

[0082] Figure 3Flow schematic of the milling machine dynamic image enhancement method provided by the embodiments of the present application Figure 1 ;

[0083] Figure 4 Schematic diagram of the panoramic view of the transparent chassis of the milling machine provided by the embodiments of the present application;

[0084] Figure 5 Schematic diagram of the dynamically enhanced image provided by the embodiments of the present application;

[0085] Figure 6 Flow schematic of the milling machine dynamic image enhancement method provided by the embodiments of the present application Figure 2 ;

[0086] Figure 7 Cross-sectional schematic diagram of the milling machine during operation provided by the embodiments of the present application;

[0087] Figure 8 Flow schematic of the milling machine dynamic image enhancement method provided by the embodiments of the present application Figure 3 ;

[0088] Figure 9 Structural schematic diagram of the milling machine dynamic image enhancement device provided by the embodiments of the present application;

[0089] Figure 10 Structural schematic diagram of the milling machine dynamic image enhancement system provided by the embodiments of the present application;

[0090] Figure 11 Structural schematic diagram of the electronic device provided by the embodiments of the present application.

[0091] Through the above-mentioned drawings, the specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed Description of the Embodiments

[0092] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of the devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0093] First, the terms involved in the present application will be explained:

[0094] Milling drum: It refers to a key component in a milling machine, usually a cylindrical part with multiple cutters or tool heads on its surface. These cutters can be made of cemented carbide or other wear-resistant materials. When the milling machine is working, the milling drum rotates at high speed driven by power, and the cutters contact the road surface and cut the material, thus removing the old road surface layer. The design and configuration of the milling drum can be adjusted according to different construction requirements, such as changing the arrangement or spacing of the cutters to adapt to different road materials and construction thicknesses.

[0095] Figure 1 Schematic diagram of the scene when the milling machine provided by the embodiment of the present application is operating Figure 1 As Figure 1 shown, in the specific application scenario of the embodiment of the present application, the milling machine performs milling operations on the working surface, and there are prohibited milling areas on the working surface, such as manhole covers, bridge expansion joints, etc. One or more cameras and one or more video display devices are arranged on the body of the milling machine. The working surface can be photographed by the camera, and the road surface conditions can be displayed through the video display device, so as to facilitate the operator to control the operation of the milling machine. The slanted lined area is the area that has been milled by the milling machine.

[0096] Figure 2 Schematic diagram of the scene when the milling machine provided by the embodiment of the present application is operating Figure 2 As Figure 2 shown, on the basis of the Figure 1 embodiment, this embodiment shows that as the milling machine operates, the manhole cover is blocked by the body of the milling machine. At this time, the camera cannot photograph the blocked manhole cover, resulting in the operator being unable to directly see the manhole cover or other prohibited milling areas, and also unable to intuitively observe the distance between the milling drum and these areas. In this case, the operator needs to rely on the command of other operators to avoid milling the prohibited milling areas.

[0097] Combined with the above scenarios, in the prior art, the common practice is for the operators on the ground to command the operators of the milling machine through gestures or other signals to lift the milling drum when approaching the prohibited milling area, or lower the milling drum after passing and then continue the operation. This method relies on manual command, so it is prone to problems of poor communication. Especially in a noisy construction environment, gestures or sound signals may not be clear enough, resulting in the operator being unable to accurately receive the instructions, increasing the risk of misoperation. And due to the limited vision of the operator, and there is usually a certain reaction time for manual command, the operator may not be able to react in time, resulting in the milling drum not being lifted or lowered in time, which may damage the manhole cover and other prohibited milling areas or the milling machine, while increasing the demand and cost of human resources and having low operation efficiency.

[0098] The dynamic image enhancement method for a milling machine provided by this application collects the forward working surface image and the side image of the milling machine, that is, all necessary visual data is collected to provide a basis for subsequent image processing and analysis. On this basis, according to the operating parameters of the milling drum on the milling machine and the side image, the first image of the milling drum is determined, thereby generating an image containing key position information for subsequent image superposition. In addition, the target distance between the prohibited milling target and the target acquisition device is calculated, and the panoramic view of the transparent chassis of the milling machine and the second position of the prohibited milling target are determined, so as to determine the relative position between the prohibited milling target and the cutting line of the milling drum. For the operator, this relative position determines the forward distance of the milling machine and the cutting position of the milling drum during milling, so that the risk of accidentally milling the prohibited milling area can be avoided, thereby reducing the risk of equipment damage and construction accidents. After that, the first image, the second position of the prohibited milling target, and the relative position are superimposed on the panoramic view of the transparent chassis to generate a dynamic enhanced image. This image can be updated in real time and provided to the operator to provide enhanced visual feedback, enabling the operator to obtain real-time information about the working surface under the milling machine and intuitively understand the specific situation of the working surface and potential obstacles through the enhanced image, which not only improves the efficiency of the milling operation but also reduces the safety hazards caused by blocked vision.

[0099] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0100] Figure 3 Schematic flow of the dynamic image enhancement method for a milling machine provided by an embodiment of this application Figure 1 As Figure 3 shown, this method includes:

[0101] S101. Obtain the forward working surface image and the side image of the milling machine.

[0102] Among them, the forward working surface image can refer to the road surface image to be processed in the forward route of the milling machine, which can help the operator understand the construction environment and potential obstacles ahead.

[0103] The side image can refer to the image of the working environment where the milling machine is located except for the road surface to be processed in the forward route, including the road surface and the body of the milling machine, which can provide key information about the height, angle, and position of the milling drum for observing the working state and position of the milling drum.

[0104] In this embodiment, the forward working surface image and the side image of the milling machine can capture image data from different perspectives through target acquisition devices installed on the milling machine, such as cameras and sensors. Further, to obtain a more complete view, multiple target acquisition devices can be respectively set at the front and side of the milling machine. The acquired image data can be transmitted to a processing unit, such as a computing platform, for subsequent image processing and analysis.

[0105] S102. Determine the first image of the milling drum based on the operating parameters of the milling drum on the milling machine and the side image.

[0106] Among them, the first image includes the first position of the cutting line of the milling drum in the side image and the projection position of the milling drum.

[0107] The operating parameters of the milling drum include the working depth, angle, rotational speed, etc. of the milling drum. These parameters determine the specific state and position of the milling drum during operation. They can be monitored and recorded in real time through the control system of the milling machine, and further calculations can be carried out in combination with the factory-set parameters of the milling machine.

[0108] In this embodiment, based on the operating parameters of the milling drum and the side image, the cutting line and projection position of the milling drum in the current operating state can be calculated, and through image processing technology, the calculated cutting line and projection position can be superimposed on the side image, so as to integrate this information into one image to form the first image, which is used for subsequent steps of superimposing and analyzing with other image data.

[0109] It can be understood that by combining the operating parameters with the side image, the actual position and cutting path of the milling drum during operation can be accurately determined, helping the operator to better control the milling process and avoid misoperation.

[0110] S103. Determine the panoramic view of the transparent chassis of the milling machine, the target distance between the milling-prohibited target and the target acquisition device, and the second position of the milling-prohibited target based on the forward working surface image and the side image of the milling machine.

[0111] Among them, the panoramic view of the transparent chassis is a processed virtual image that shows the panoramic view of the milling machine chassis. Although called "transparent", it actually refers to an image generated through technical means such as computer image processing, enabling the operator to "see" the situation under the milling machine chassis. This image provides an overall perspective to help the operator understand the position and posture of the milling machine in the operating environment.

[0112] As Figure 4 shown, Figure 4Schematic diagram of the panoramic view of the transparent chassis of the milling machine provided by the embodiment of the present application. In this embodiment, the panoramic view of the transparent chassis of the milling machine is formed by splicing the ground projection of the milling machine chassis area and the surrounding images of the milling machine body. The surrounding images of the milling machine body can be spliced according to the forward working surface image and the side image of the milling machine.

[0113] In this embodiment, the movement trajectory of the milling machine can be calculated according to the historical travel direction information and the corresponding travel distance information of the milling machine, and combined with the engineering parameters of the milling machine to obtain the panoramic historical frame projection image, which is the ground projection of the milling machine chassis area.

[0114] The milling prohibited target can refer to the area that needs to be avoided during the milling operation, such as manhole covers, bridge expansion joints, etc., to prevent equipment damage or construction quality problems.

[0115] The target acquisition device can refer to cameras or sensors installed on the milling machine, etc., for capturing images of the working environment.

[0116] In this embodiment, image processing technology can be used to analyze the forward working surface image and the side image to identify the position of the milling prohibited target. At the same time, the boundary and position of the milling prohibited target can be detected through image feature recognition algorithms.

[0117] Furthermore, according to the identified position of the milling prohibited target, through geometric transformation and distance measurement algorithms, the target distance between it and the target acquisition device (such as a camera) is calculated.

[0118] Then, the position of the milling prohibited target is converted into a second position in the coordinate system relative to the milling machine to ensure that the position of the milling prohibited target can be used for image overlay and dynamic enhancement in subsequent steps.

[0119] It can be understood that by combining the forward working surface image and the side image, the milling prohibited target can be accurately identified and located, improving the accuracy of the milling prohibited target recognition and reducing the possibility of misrecognition. And the distance between the milling prohibited target and the milling machine is calculated in real time, enabling the operator to adjust the operation of the milling drum in time to avoid mis-milling, thus improving the flexibility and response speed of the construction process.

[0120] S104. Determine the relative position between the milling prohibited target and the cutting line according to the first position and the target distance.

[0121] The relative position between the milling prohibited target and the cutting line refers to the spatial relationship between the milling prohibited target and the cutting line generated by the milling drum during milling, including the distance and direction between them.

[0122] It should be noted that for the operator, the relative position between the milling prohibited target and the cutting line is key decision-making information for guiding the operation of the milling machine.

[0123] In this embodiment, the target distance between the first position (the cutting line position of the milling drum) and the milling prohibited target is used. Through geometric calculation and coordinate transformation, the relative position between the milling prohibited target and the cutting line is calculated. Then, the calculated relative position is converted into operable information, such as distance and direction, so that the operator or the automation system can adjust the operation of the milling machine according to this information. For example, active obstacle avoidance is performed for the milling prohibited target, including controlling the lifting of the milling machine legs, decelerating or stopping during driving, and stopping the floating of the side baffle, etc.

[0124] It can be understood that by clarifying the relative position between the milling prohibited target and the cutting line, the operator can more precisely control the operation of the milling drum, avoid mis-milling, and improve the accuracy and safety of the milling operation. On the other hand, this precise relative position information can be used in the automation control system, enabling the milling machine to autonomously adjust the operation to avoid the milling prohibited area, reducing the dependence on manual command, and improving the operation efficiency.

[0125] S105. Based on the panoramic view of the transparent chassis, the first image, the second position, and the relative position are superimposed to obtain a dynamically enhanced image.

[0126] In this embodiment, through image processing technology, information from different sources, including the information of the first image, the second position, and the relative position, can be integrated into a unified view (the panoramic view of the transparent chassis). The generated dynamically enhanced image is updated in real time, reflecting the current operation status and environment of the milling machine. This image is provided to the operator for guiding the operation of the milling machine.

[0127] As Figure 5 shown, Figure 5 is a schematic diagram of the dynamically enhanced image provided by the embodiment of the present application. Based on the Figure 4 embodiment, a dynamically enhanced image showing the projection position of the milling drum and the cutting line position is displayed. At the same time, on this image, the operator can intuitively see the relative position between the milling prohibited target and the cutting line, as well as the distance information in the advancing direction of the milling machine during the operation process. When the relative position is too close, an alarm message can be immediately generated to prompt the operator to pay attention to the operation safety.

[0128] In this embodiment, a relative position threshold can be set according to historical milling operation experience. When the relative position between the milling prohibited target and the cutting line is less than this relative position threshold, an alarm message is immediately generated.

[0129] It can be understood that through this dynamically enhanced image, the operator can observe the position of the current milling machine and the milling drum (projection diagram) in real time, and understand the position information of the areas where milling is prohibited. On this basis, the relative position between the milling drum and the cutting line and the relative position between the areas where milling is prohibited and the cutting line can also be determined. By integrating and displaying this position information, a comprehensive perspective can be provided, integrating the operating status and environmental information of the milling machine, enabling the operator to understand all key information through one image, improving the intuitiveness of operation, effectively avoiding mis-milling of the areas where milling is prohibited, thus protecting the safety of the equipment and the construction site, and improving the efficiency and quality of the milling operation.

[0130] The milling machine dynamic image enhancement method provided by the embodiment of the present application obtains the forward working surface image and the side image of the milling machine, then determines the position of the cutting line of the milling drum in the side image and the projection position of the milling drum, calculates the target distance between the areas where milling is prohibited and the target acquisition device at the same time, and determines the panoramic view of the transparent chassis of the milling machine and the second position of the areas where milling is prohibited, thereby obtaining the relative position between the areas where milling is prohibited and the cutting line of the milling drum. Through image superposition, a dynamically enhanced image is obtained, enabling the operator to more clearly see the relationship between the milling drum and the areas where milling is prohibited, reducing the problem of line of sight occlusion, avoiding mis-milling of the areas where milling is prohibited, reducing the dependence on manual command, and improving the construction efficiency.

[0131] Figure 6 It is a schematic flow chart of the milling machine dynamic image enhancement method provided by the embodiment of the present application

[0132] II. As Figure 6 shown, on the basis of the Figure 3 embodiment, steps S102 and S103

[0133] are described in detail, specifically including the following steps:

[0134] S201. Determine the diameter of the milling drum, the milling depth, and the cross-sectional information generated by the milling drum according to the operation parameters.

[0135] Among them, the diameter of the milling drum refers to the overall size of the milling drum, which affects the cutting ability and coverage of the milling drum.

[0136] The milling depth refers to the depth at which the milling drum cuts into the road surface during operation, which determines the thickness of the road surface material removed by the milling machine and is an important indicator of the construction quality and effect.

[0137] The cross-sectional information refers to the cross-sectional shape of the milling drum at a specific milling depth, which shows the geometric characteristics of the contact between the milling drum and the road surface and affects the cutting path and efficiency.

[0138] In this embodiment, according to the factory-set parameters of the milling machine or the parameters of the current operation obtained from the control system and sensors of the milling machine, the diameter and milling depth of the milling drum are determined, and then the cross-sectional information of the milling drum under the current operating conditions is calculated.

[0139] S202. Determine the chord length of the cutting line of the cross-section according to the diameter of the milling drum, the milling depth, and the cross-sectional information generated by the milling drum.

[0140] In this embodiment, the diameter and milling depth of the milling drum can be converted into geometric calculations to determine the specific shape of the contact between the milling drum and the road surface. Then, by analyzing the cross-sectional shape, the chord length of the cutting line of the contact between the milling drum and the road surface is calculated.

[0141] As Figure 7 shown, Figure 7 is a schematic cross-sectional view of the milling machine during operation provided by the embodiment of the present application. In this embodiment, the center of the milling drum is O, the radius is R, and the diameter is 2R. During the milling process, the contact interval between the milling drum and the ground is between a and b, and the milling depth is H. Thus, according to the Pythagorean theorem, the chord length in the cross-sectional information generated by the milling drum is 2W. This calculation process involves basic geometry content and is common knowledge in the art, so it will not be elaborated in this embodiment.

[0142] S203. Determine the marked points on the milling drum housing in the side image according to the side image.

[0143] Among them, the marked points are specific points on the milling drum housing, which are used to identify the center position of the milling drum at different milling depths. Usually, they are visible marks or structural features for easy identification in the image.

[0144] In this embodiment, the side image is analyzed by image processing technology to identify the marked points on the milling drum housing, and the depth correspondence relationship is determined by combining the factory-set parameters of the milling machine or the pre-calibrated data. Thus, the center position of the milling drum at different depths can be inferred from the position of the marked points.

[0145] Exemplarily, edge detection, shape recognition, or other image processing algorithms can be used to accurately locate the marked points.

[0146] S204. Map the chord length of the cutting line of the cross-section and the milling drum to the side image according to the marked points to determine the first image of the milling drum.

[0147] In this embodiment, after calculating the chord length of the cutting line and determining the landmark points, according to the positions of the landmark points, the chord length of the cutting line of the cross-section can be mapped to the side image through geometric transformation or image superposition. Thus, the cutting path and position of the milling drum are integrated into the side image, generating a first image containing this information, which can be used for superposition and analysis with other image data in subsequent steps.

[0148] It can be understood that by mapping the chord length of the cutting line and the position of the milling drum to the side image, an accurate visual representation of the milling drum during operation is provided, improving the intuitiveness and accuracy of operation. As key visual data, the first image provides basic support for subsequent image superposition and dynamic enhancement, ensuring the coherence and effectiveness of the entire method process.

[0149] S205. Obtain a panoramic image of the milling machine and a panoramic image of the transparent chassis of the milling machine based on the forward working face image and the side image of the milling machine.

[0150] Among them, the panoramic image refers to a comprehensive view integrated from the forward working face image and the side image, providing an overall perspective to help the operator understand the position and state of the milling machine in the working environment.

[0151] In this embodiment, the forward working face image and the side image are obtained from a camera installed on the milling machine, and image stitching or fusion techniques, such as geometric correction, color adjustment, and edge processing, are used to integrate the forward working face image and the side image into a panoramic image and ensure seamless integration of the images. The generated panoramic image provides a comprehensive perspective.

[0152] The panoramic image of the transparent chassis of the milling machine is formed by stitching the ground projection of the milling machine chassis area and the image around the milling machine body, which has been described in detail in the above embodiment and will not be elaborated in this embodiment.

[0153] S206. Perform target recognition processing on the panoramic image to obtain an initial image, which contains the bounding box of the prohibited milling target and the corresponding first category label.

[0154] Among them, target recognition processing can refer to using computer vision technology to identify specific objects or regions in an image, usually involving two stages: detecting the position of the object (bounding box) and classifying the type of the object (category label). In this embodiment, the bounding box is used to identify the position of the prohibited milling target in the panoramic image. The first category label is used to describe the type of the prohibited milling target, such as manhole covers, bridge expansion joints, etc.

[0155] As an example, first, preprocess the panoramic image, such as denoising, enhancing contrast, etc., to improve the accuracy of target recognition. Then, use object detection algorithms (such as YOLO, Faster R-CNN, etc.) to identify potential milling-prohibited targets in the panoramic image and generate a bounding box for each target, marking its position in the image. After that, classify the detected targets, assign corresponding first-category labels, and then use a classification algorithm or a pre-trained neural network model to classify the target objects into specific categories according to features. Finally, overlay the bounding boxes and category labels of the identified milling-prohibited targets onto the panoramic image to generate an initial image.

[0156] It can be understood that through object recognition processing, it is possible to accurately identify the milling-prohibited targets that need to be avoided during the milling operation, improve the accuracy of recognition, reduce the possibility of misrecognition, and the real-time generated initial image can help the operator quickly identify and avoid the milling-prohibited targets, improving the flexibility and response speed of the construction process.

[0157] S207. Perform semantic segmentation processing on the initial image to obtain the second-category label corresponding to each pixel in the initial image.

[0158] Among them, semantic segmentation refers to a computer vision technology used to assign each pixel in an image to a category. Different from object recognition, semantic segmentation provides a more fine-grained classification result. In this embodiment, semantic segmentation can identify the categories of different regions in the image, such as the road surface, obstacles, milling-prohibited targets, etc. The second-category label can refer to the category label assigned to each pixel during semantic segmentation processing, used to describe the category to which the pixel belongs.

[0159] As another example, first perform necessary preprocessing on the initial image, such as normalization, size adjustment, etc., to meet the input requirements of the semantic segmentation model. Then, use a pre-trained semantic segmentation model (such as U-Net, DeepLab, etc.) to process the initial image, classify each pixel in the image, generate the second-category label for each pixel, and overlay the segmentation result onto the initial image to generate an image containing the category information of each pixel, thereby providing a more fine-grained understanding of the environment.

[0160] It can be understood that through semantic segmentation, it is possible to identify different regions and objects in the image at the pixel level, providing more detailed environmental information than object recognition, supporting more precise operation control. In addition, semantic segmentation can identify more details, reduce the possibility of misrecognition, and improve the adaptability to complex environments, especially in the case of multiple targets and complex backgrounds.

[0161] S208. Determine the second position of the milling-prohibited target according to the bounding box, the first-category label, and the second-category label.

[0162] Among them, the second position may refer to the precise position of the milling - prohibited target in the operating environment determined after comprehensively analyzing the bounding box and the class label.

[0163] In this embodiment, by using the preliminary position provided by the bounding box and the information of the first - class label, a potential milling - prohibited target area is identified. Combining with the second - class label of semantic segmentation, the position of the milling - prohibited target is further refined. Inside the bounding box, by analyzing the second - class label, the specific boundary and shape of the milling - prohibited target are determined. Finally, the analysis results are integrated to determine the second position of the milling - prohibited target.

[0164] S209. Determine the target distance between the milling - prohibited target and the target acquisition device according to the milling depth and the second position.

[0165] In this embodiment, by quantifying the distance between each object, using the second position of the milling - prohibited target determined in the previous step, its precise coordinates in the operating environment are obtained, and according to the current state of the milling machine and the milling depth, the position of the target acquisition device in the operating environment is determined, and then the target distance between the milling - prohibited target and the target acquisition device is calculated.

[0166] As Figure 7 shown, in this embodiment, the milling depth is H; the distance between the target acquisition device and the center of the milling drum is P1; according to the position of the milling - prohibited target, the target distance D0 between the milling - prohibited target and the target acquisition device can be determined; according to the calculation of each parameter information, the distance D1 between the milling - prohibited target and the center of the milling drum is D1 = D0+P1.

[0167] In a possible implementation manner, the specific implementation of S209 can be through the following steps:

[0168] First, according to the milling depth, determine the calibration parameters of the target acquisition device; then, according to the calibration parameters, convert the image coordinate system into a space coordinate system; and then, based on the space coordinate system, according to the second position, determine the target distance between the milling - prohibited target and the target acquisition device.

[0169] Among them, the calibration parameters include the vertical height and angle of the target acquisition device relative to the ground, which are used to describe the position and orientation of the target acquisition device (such as a camera) in space.

[0170] The image coordinate system refers to the pixel position in the image, usually represented by two - dimensional coordinates, and the space coordinate system is a three - dimensional coordinate system used to describe the position of an object in the actual space. In this embodiment, the origin of the space coordinate system is set as the position of the milling machine.

[0171] In this embodiment, through geometric transformation and calibration, using the calibration parameters, the position of the milling - prohibited target in the image coordinate system can be converted into the position in the space coordinate system to ensure that the image information is accurately mapped to the real space. Then, based on the position of the milling - prohibited target in the space coordinate system, the target distance between it and the target acquisition device is calculated.

[0172] It should be noted that the target distance refers to the distance between the milling - prohibited target and the target acquisition device on the working surface of the milling machine. Usually, the working surface of the milling machine is a horizontal plane, so the target distance is usually the horizontal distance between the milling - prohibited target and the target acquisition device. However, when the working surface is a non - horizontal plane, for example, when the milled road surface is an uphill road surface, the target distance needs to consider the change in the vertical direction, that is, the target acquisition device is projected onto the working surface. At this time, the target distance is the straight - line distance between the milling - prohibited target and the target acquisition device.

[0173] It can be understood that through precise coordinate transformation and distance calculation, an accurate target distance between the milling - prohibited target and the target acquisition device is provided, reducing the measurement error caused by the changes in the position and angle of the device.

[0174] In this embodiment, the target acquisition device is set at the front - left position and the front - right position of the milling machine, and is used to acquire the forward working - face image and the side image of the milling machine.

[0175] As an example, the target acquisition devices can include six, which are respectively arranged at the front left position, the front middle position, the front right position of the milling machine, the left position, the right position of the fuselage of the milling machine, and the rear position. Among them, the target acquisition device at the front left position is used to obtain the working surface image in the front left of the milling machine, mainly for monitoring the working state and target area in the front left of the milling machine to ensure the accuracy of milling, especially when driving on the left side. The device at the front middle position is used to obtain the working surface image in the middle front of the milling machine, focusing on monitoring the main working area in the front of the milling machine to help observe the milling depth, milling quality, and whether there are abnormal conditions. The device at the front right position is used to obtain the working surface image in the front right of the milling machine. Similar to the front left device, it is responsible for monitoring the working state in the front right of the milling machine to ensure the overall uniformity and no omission of the working surface. The device at the left position of the fuselage is used to obtain the side image on the left side of the milling machine, mainly monitoring the operation progress on the left side, the road surface condition, and the contact situation between the device and the road surface during the working process of the milling machine to help with the lateral operation quality control. The device at the right position of the fuselage is used to obtain the side image on the right side of the milling machine. Similar to the left device, it monitors the operation area on the right side of the milling machine to ensure the flatness and quality of the operation and avoid lateral deviation. The device at the rear position is used to obtain the image behind the milling machine, mainly monitoring the progress of the subsequent operation of the milling machine to ensure the integrity of the rear operation, checking the road surface effect left after the milling machine operates, and the driving track of the milling machine. These six target acquisition devices are respectively arranged at different positions, which can comprehensively monitor the operation conditions of the front, side, and rear of the milling machine to ensure operation accuracy, quality control, and operation safety.

[0176] It can be understood that through multi-angle image acquisition, the operator can obtain a comprehensive perspective to understand the overall situation of the milling machine in the operating environment. This comprehensive perspective helps to better judge the relative position between the milling machine and the road surface and obstacles, and provides detailed visual information, enabling the operator to more accurately control the movement of the milling machine and the operation of the milling drum, reducing the risk of misoperation caused by limited vision.

[0177] The milling machine dynamic image enhancement method provided by the embodiments of the present application can accurately identify the position and category of the prohibited milling target through relevant calculations based on operation parameters and through detailed image processing and analysis. The obtained accurate position information and cutting path can help the operator avoid mis-milling the prohibited milling area. At the same time, the dynamically updated image and position information enable the operator to quickly adjust the operation of the milling machine, improving the flexibility of the construction process.

[0178] Figure 8 Schematic flow of the milling machine dynamic image enhancement method provided by the embodiments of the present application Figure 3 As Figure 8As shown, in this embodiment, based on the Figure 3 embodiment, step S104 is described in detail, which specifically includes the following steps:

[0179] S301. Determine the milling speed of the milling machine according to the operation parameters.

[0180] In this embodiment, the operation parameters may refer to various factors affecting the operation of the milling machine, including but not limited to: road surface materials and conditions (such as hardness, humidity), current milling depth, load conditions, environmental conditions (such as temperature, humidity), equipment status (such as wear degree, maintenance status), etc. According to the collected operation parameters, the current operation conditions can be analyzed to determine the milling speed.

[0181] S302. Determine the first dynamic position of the milling prohibited target according to the milling speed and the target distance.

[0182] Among them, the first dynamic position may refer to the position of the milling prohibited target at a future moment predicted according to the current milling speed and the initial position of the milling prohibited target.

[0183] In this embodiment, after determining the milling speed, according to the current milling speed, the initial position of the milling prohibited target, and the target distance between the milling prohibited target and the target acquisition device obtained in the above embodiment, the dynamic position at a future moment can be calculated using conventional kinematic logic, for example, using speed and time interval to predict the position change.

[0184] It should be noted that the actual position of the milling prohibited target on the road surface does not change. The position changes and dynamic positions (including the first dynamic position and the second dynamic position) involved in this embodiment all represent the position of the milling prohibited target relative to the milling machine.

[0185] S303. When the milling prohibited target is not located at the bottom of the milling machine, determine the relative position between the milling prohibited target and the cutting line according to the first position and the first dynamic position.

[0186] In this embodiment, based on the first position of the cutting line in the image and the predicted first dynamic position of the milling prohibited target, the shortest distance between the milling prohibited target and the cutting line and the direction of the milling prohibited target relative to the cutting line can be calculated using geometric methods.

[0187] It can be understood that when the milling prohibited target has not entered the direct operation area of the milling machine, its relative position is evaluated in advance to facilitate effective path planning and avoidance.

[0188] S304. When the milling prohibited target is located at the bottom of the milling machine, determine the predicted position of the milling prohibited target according to the second position.

[0189] In this embodiment, the second position of the milling prohibited target is obtained from the previous steps, which is the exact position of the current milling prohibited target at the bottom of the milling machine. Then, by analyzing the current motion state (such as speed and direction) and environmental conditions of the milling prohibited target, its future motion trend is evaluated to obtain the predicted position of the milling prohibited target.

[0190] Exemplarily, a kinematic model or prediction algorithm can be used to calculate the predicted position of the milling prohibited target according to the current second position and motion trend through time series prediction, machine learning models or simple physical motion models.

[0191] S305. Correct the first dynamic position according to the predicted position to obtain the second dynamic position of the milling prohibited target.

[0192] Among them, the second dynamic position can refer to the dynamic position of the milling prohibited target after correction processing, which can provide more accurate position information.

[0193] In this embodiment, by comparing the difference between the predicted position and the first dynamic position, possible error sources (such as speed changes, environmental impacts, etc.) are analyzed, and the position difference information is used to correct the first dynamic position by methods such as weighted average or filtering algorithms, and the corrected result is used as the second dynamic position of the milling prohibited target.

[0194] S306. Determine the relative position between the milling prohibited target and the cutting line according to the second dynamic position.

[0195] In this embodiment, the process of determining the relative position is the same as that in the above embodiment, and will not be elaborated here.

[0196] As Figure 7 shown, in this embodiment, the relative position Y between the milling prohibited target and the cutting line is Y = D1 - W.

[0197] It can be understood that when the milling prohibited target has entered the direct operation area of the milling machine and is completely blocked by the vehicle body, by further predicting its future position and performing correction processing, more accurate avoidance and path adjustment can be carried out.

[0198] The milling machine dynamic image enhancement method provided by the embodiments of the present application calculates the actual milling speed of the milling machine according to operation parameters (such as current load, road surface conditions, etc.), thereby determining the relative position change between the milling machine and the milling prohibited target during operation. Through dynamic position calculation and correction, the tracking accuracy of the milling prohibited target under dynamic conditions is improved, ensuring accuracy under different speed and position conditions. At the same time, accurate target positioning and path planning improve the efficiency and quality of the milling operation, and reduce unnecessary downtime and adjustment time.

[0199] Figure 9This is a schematic structural diagram of the dynamic image enhancement device for a milling machine provided by an embodiment of the present application. As Figure 9 shown, the dynamic image enhancement device 40 for a milling machine provided in this embodiment includes:

[0200] An acquisition module 401, configured to acquire the forward working surface image and the side image of the milling machine;

[0201] A first determination module 402, configured to determine a first image of the milling drum according to the operation parameters of the milling drum on the milling machine and the side image, where the first image includes the first position of the cutting line of the milling drum in the side image and the projection position of the milling drum;

[0202] A second determination module 403, configured to determine the panoramic view of the transparent chassis of the milling machine, the target distance between the milling prohibited target and the target acquisition device, and the second position of the milling prohibited target according to the forward working surface image and the side image of the milling machine;

[0203] A third determination module 404, configured to determine the relative position between the milling prohibited target and the cutting line according to the first position and the target distance;

[0204] A processing module 405, configured to superimpose the first image, the second position, and the relative position based on the panoramic view of the transparent chassis to obtain a dynamically enhanced image.

[0205] In a possible implementation manner, the first determination module 402 is specifically configured to:

[0206] Determine the diameter, milling depth, and cross-sectional information generated by the milling drum according to the operation parameters;

[0207] Determine the chord length of the cutting line of the cross-section according to the diameter, milling depth, and cross-sectional information of the milling drum;

[0208] Determine the marking points on the milling drum housing in the side image according to the side image, where the marking points represent the center positions of the milling drums corresponding to different milling depths;

[0209] Map the chord length of the cutting line of the cross-section and the milling drum to the side image according to the marking points to determine the first image of the milling drum.

[0210] In a possible implementation manner, the second determination module 403 is specifically configured to:

[0211] Obtain the panoramic image of the milling machine and the panoramic view of the transparent chassis of the milling machine according to the forward working surface image and the side image of the milling machine;

[0212] Perform target recognition processing on the panoramic image to obtain an initial image, where the initial image includes the bounding box of the milling prohibited target and the corresponding first category label;

[0213] Perform semantic segmentation on the initial image to obtain the second category label corresponding to each pixel in the initial image;

[0214] Determine the second position of the milling-prohibited target according to the bounding box, the first category label, and the second category label;

[0215] Determine the target distance between the milling-prohibited target and the target acquisition device according to the milling depth and the second position.

[0216] In a possible implementation manner, the second determination module 403 is further configured to:

[0217] Determine the calibration parameters of the target acquisition device according to the milling depth, where the calibration parameters include the vertical height and angle of the target acquisition device relative to the ground;

[0218] Convert the image coordinate system to a space coordinate system according to the calibration parameters, where the origin of the space coordinate system is the milling machine;

[0219] Based on the space coordinate system, determine the target distance between the milling-prohibited target and the target acquisition device according to the second position.

[0220] In a possible implementation manner, the target acquisition device is arranged at the front left position and the front right position of the milling machine, and is used to acquire the forward working surface image and the side image of the milling machine.

[0221] In a possible implementation manner, the third determination module 404 is specifically configured to:

[0222] Determine the milling speed of the milling machine according to the operation parameters;

[0223] Determine the first dynamic position of the milling-prohibited target according to the milling speed and the target distance;

[0224] When the milling-prohibited target is not located at the bottom of the milling machine, determine the relative position between the milling-prohibited target and the cutting line according to the first position and the first dynamic position.

[0225] In a possible implementation manner, the third determination module 404 is further configured to:

[0226] When the milling-prohibited target is located at the bottom of the milling machine, determine the predicted position of the milling-prohibited target according to the second position;

[0227] Perform correction processing on the first dynamic position according to the predicted position to obtain the second dynamic position of the milling-prohibited target;

[0228] Determine the relative position between the milling-prohibited target and the cutting line according to the second dynamic position.

[0229] The milling machine dynamic image enhancement device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effects are similar. Details are not described herein in this embodiment.

[0230] Figure 10 It is a schematic structural diagram of the milling machine dynamic image enhancement system provided in an embodiment of the present application. As Figure 10 shown, the milling machine dynamic image enhancement system 50 provided in this embodiment includes: a frame 501, a milling drum 502, a milling drum housing 503, a target acquisition device 504, an electronic device 505, and a video display device 506. Among them:

[0231] The frame 501 is supported on a plurality of traction devices and travels along the working surface, parallel to the traveling axis of the milling machine;

[0232] The milling drum 502 is rotatably supported on the frame 501 and is used for milling the working surface;

[0233] The milling drum housing 503 is supported on the frame 501 and is used to accommodate the milling drum 502;

[0234] The target acquisition device 504 is used to acquire the forward working surface image and the side image of the milling machine;

[0235] The electronic device 505 is used to execute the above method embodiment;

[0236] The video display device 506 is used to display the dynamically enhanced image.

[0237] The milling machine dynamic image enhancement system provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effects are similar. Details are not described herein in this embodiment.

[0238] Figure 11 It is a schematic structural diagram of the electronic device provided in an embodiment of the present application. As Figure 11 shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. Among them, the processor 601, the memory 602, and the communication component 603 are connected through a bus 604.

[0239] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.

[0240] For the specific implementation process of the processor 601, reference can be made to the above method embodiment, and its implementation principle and technical effects are similar. Details are not described herein in this embodiment.

[0241] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU for short), or may also be other general-purpose processors, digital signal processors (DSP for short), application specific integrated circuits (ASIC for short), etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by the execution of the hardware processor, or can be implemented by the combination of hardware and software modules in the processor.

[0242] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0243] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0244] This application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0245] This application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the processor executes the computer-executable instructions, the above method is implemented.

[0246] The above-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk or an optical disk. The readable storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0247] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in a device.

[0248] The division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed between each other can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0249] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0250] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0251] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical discs that can store program codes.

[0252] Those of ordinary skill in the art will understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the aforementioned storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0253] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include known common knowledge or conventional technical means in the technical field not disclosed by the present invention. It is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A method for dynamic image enhancement of a milling machine, characterized in that: include: Acquire the forward working surface image and the side image of the milling machine; Determine a first image of the milling drum according to the operating parameters of the milling drum on the milling machine and the side image, wherein the first image includes a first position of a cutting line of the milling drum in the side image and a projection position of the milling drum; Determine, according to the forward working surface image and the side image of the milling machine, a panoramic image of the transparent chassis of the milling machine, a target distance between a prohibited milling target and a target acquisition device, and a second position of the prohibited milling target; Determining the relative position between the prohibited milling target and the cutting line according to the first position and the target distance; Based on the transparent chassis panoramic image, the first image, the second position, and the relative position are superimposed to obtain a dynamic enhanced image.

2. The method according to claim 1, characterized in that Determining the first image of the milling drum according to the operating parameters of the milling drum on the milling machine and the side image includes: Determine the diameter of the milling drum, the milling depth, and the cross-section information generated by the milling drum according to the operation parameters; Determine the chord length of the cutting line of the cross section according to the diameter of the milling drum, the milling depth, and the cross section information generated by the milling drum; According to the side image, determining a mark point on a milling drum cover in the side image, wherein the mark point represents a center position of the milling drum corresponding to different milling depths; According to the marking points, the chord length of the cutting line of the cross section and the milling drum are mapped into the side image to determine a first image of the milling drum.

3. The method according to claim 2, characterized in that Determining the transparent chassis panoramic image of the milling machine, the target distance between the prohibited milling target and the target acquisition device, and the second position of the prohibited milling target according to the forward working surface image and the side image of the milling machine includes: Obtaining a panoramic image of the milling machine and a panoramic image of a transparent chassis of the milling machine according to the forward working surface image and the side image of the milling machine; Performing target recognition processing on the panoramic image to obtain an initial image, wherein the initial image includes a bounding box of the prohibited milling target and a corresponding first category label; Performing semantic segmentation processing on the initial image to obtain a second category label corresponding to each pixel in the initial image; Determining a second position of the prohibited milling target according to the bounding box and the first category label and the second category label; A target distance between the prohibited milling target and the target acquisition device is determined according to the milling depth and the second position.

4. The method according to claim 3, characterized in that Determining the target distance between the prohibited milling target and the target acquisition device according to the milling depth and the second position includes: Determining calibration parameters of the target acquisition device according to the milling depth, wherein the calibration parameters include a vertical height and an angle of the target acquisition device relative to the ground; According to the calibration parameters, the image coordinate system is converted into a spatial coordinate system, the origin of which is the milling machine; Based on the spatial coordinate system and according to the second position, a target distance between the prohibited milling target and the target acquisition device is determined.

5. The method according to claim 4, characterized in that The target acquisition device is arranged at the front left position and the front right position of the milling machine, and is used to obtain the forward working surface image and the side image of the milling machine.

6. The method according to any one of claims 1 to 5, characterized in that Determining the relative position between the prohibited milling target and the cutting line according to the first position and the target distance includes: Determining the milling speed of the milling machine according to the operating parameters; Determining a first dynamic position of the prohibited milling target according to the milling speed and the target distance; When the prohibited milling target is not located at the bottom of the milling machine, the relative position between the prohibited milling target and the cutting line is determined according to the first position and the first dynamic position.

7. The method according to claim 6, characterized in that After determining the first dynamic position of the prohibited milling target according to the milling speed, the method further includes: When the prohibited milling target is located at the bottom of the milling machine, determining the predicted position of the prohibited milling target according to the second position; According to the predicted position, the first dynamic position is corrected to obtain a second dynamic position of the prohibited milling target; The relative position between the prohibited milling target and the cutting line is determined according to the second dynamic position.

8. A milling machine dynamic image enhancement device, characterized in that: include: An acquisition module, used for acquiring an image of the forward working surface and a side image of the milling machine; A first determining module, configured to determine a first image of the milling drum according to an operating parameter of the milling drum on the milling machine and the side image, wherein the first image includes a first position of a cutting line of the milling drum in the side image and a projection position of the milling drum; A second determination module is used to determine a panoramic image of a transparent chassis of the milling machine, a target distance between a prohibited milling target and a target acquisition device, and a second position of the prohibited milling target according to the forward working surface image and the side image of the milling machine; a third determination module, configured to determine a relative position between the prohibited milling target and the cutting line according to the first position and the target distance; A processing module is used to superimpose the first image, the second position, and the relative position based on the transparent chassis panoramic image to obtain a dynamic enhanced image.

9. A milling machine dynamic image enhancement system, characterized in that: include: Frame, milling drum, milling drum housing, target acquisition equipment, electronic equipment, video display device; The frame is supported on a plurality of traction devices and travels along the working surface, parallel to the travel axis of the milling machine; The milling drum is rotatably supported on the frame and is used for milling the working surface; The milling drum housing is supported on the frame and is used to accommodate the milling drum; The target acquisition device is used to obtain the forward working surface image and the side image of the milling machine; The electronic device is used to execute the method according to any one of claims 1 to 7 above; The video display device is used to display dynamic enhanced images.

10. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

Images