Excavator remote construction equipment auxiliary display method and system, storage medium and excavator

Abstract

The invention discloses an excavator remote construction equipment auxiliary display method and system, a storage medium and an excavator. The method comprises the steps that a main view angle image of the excavator is shot in real time; identifying a target identification object in each frame of image, and extracting pixel coordinates; comparing the pixel coordinates one by one, performing noise reduction processing on the pixel coordinate data according to the continuous change trend of the pixel coordinates, and outputting the filtered pixel coordinates; positioning the identification position of a target identification object in the main view angle image frame by frame, and obtaining a target identification image; and amplifying the target recognition image, and superposing the target recognition image into the main view angle image in a picture-in-picture form. According to the scheme, the surrounding environment and operation details can be conveniently observed at the same time, and the data processing efficiency is high.

Description

Technical Field

[0001] The present invention relates to the field of remote control of excavators, and in particular to a method, system, storage medium and excavator for remote construction tool auxiliary display of an excavator. Background Art

[0002] Remote-controlled excavators are widely used in various occasions such as disaster relief and mining. Remote-controlled excavators mainly transmit images and sounds around the excavator through medium-range wireless technology or 5G technology. For example, the Chinese invention patent with publication number CN110747930A provides an excavator remote control system based on a 5G network, including a construction site part and a remote control room part. The two parts transmit signals through two 5G network transmission modules, which not only ensures the safety of the driver, but also thoroughly improves the driver's working environment. The audio-visual information acquisition module of the remote control system collects images and sounds of the surrounding environment through cameras and microphones, and transmits the images and sounds to the audio-visual information display module in the remote control room in real time through the 5G network transmission module. Among them, the image is mainly a full-view image obtained by cameras set around the excavator. Although the excavator can also be controlled through the full-view image, it is impossible to observe the details of the working device and it is difficult to obtain fine operation.

[0003] The above problem can be solved by enlarging the image of the excavator's equipment. However, in the process of real-time image recognition, pixel jitter is prone to occur in the image acquisition of local areas, resulting in more obvious jitter in the enlarged image of the equipment. Operators are prone to dizziness, which leads to inaccurate operation judgments and affects the safe operation of the equipment. Summary of the invention

[0004] Therefore, in order to solve the above problems, the present invention provides an excavator remote construction tool auxiliary display method, system, storage medium and excavator.

[0005] The present invention is achieved through the following technical solutions: The method for auxiliary display of remote construction tools for excavators comprises the following steps: Real-time capture of the main perspective image of the excavator during its operation; Identify the target object in each frame of the primary view image, and extract the pixel coordinates of the target object in each frame; Compare the pixel coordinates of the target identification object one by one, perform noise reduction processing on the pixel coordinate data according to the continuous change trend of the pixel coordinates, filter out the image frames with pixel jitter, and output the filtered pixel coordinates; Locate the recognition position of the target object in the primary view image frame by frame according to the filtered pixel coordinates, and obtain the target recognition image located at the recognition position; The target recognition image is enlarged and superimposed on the main viewing angle image in the form of picture-in-picture, and the main viewing angle image and its target recognition image are displayed synchronously.

[0006] Preferably, the “comparing the pixel coordinates of the target identification object one by one, performing noise reduction processing on the pixel coordinate data according to the continuous change trend of the pixel coordinates, filtering out the image frames with pixel jitter, and outputting the filtered pixel coordinates” means: Sequentially obtain the pixel coordinates of the target object in each frame of the main view image, and extract the position data in the pixel coordinates; Compare the position data of the current pixel coordinates with the position data of the previous pixel coordinates one by one, and record the comparison results. If the current position data is greater than the previous position data, it is counted as a positive number, and negative numbers are cleared to zero; if the current position data is less than or equal to the previous position data, it is counted as a negative number, and positive numbers are cleared to zero; Set a counting threshold N. When the number of records with the same comparison result is greater than the counting threshold N, output the current position data. Otherwise, discard the current data and maintain the historical data.

[0007] Preferably, the position data of the pixel coordinates include x-axis data and y-axis data, and the filtered position data are respectively obtained, and the recognition position of the target object in the main perspective image is located frame by frame through each set of position data, and the image of the recognition position is used as the target recognition image.

[0008] Preferably, the counting threshold N is the level of the filter.

[0009] Preferably, the "real-time shooting of the main perspective image of the excavator" means: a front-facing high-definition camera with the same driving perspective is set directly in front of the excavator cab, and the front-facing high-definition camera is used to shoot a wide-angle area in front of the cab, and ensure that the target identification object is always within the shooting visible range.

[0010] Preferably, the target identification object is an excavator tool.

[0011] Excavator remote construction tool auxiliary display system, including: A camera module is used to capture the main perspective image of the excavator in real time during the working process of the excavator; A recognition module, used to obtain a target object of the excavator in the primary view image and extract pixel coordinates of the target object; A filtering module, comprising a filter, wherein the filter comprises a positive counter and a negative counter, and is used to compare the pixel coordinates of the target identification object frame by frame, perform noise reduction processing on the pixel coordinate data according to the continuous change trend of the pixel coordinates, filter out the image frames with pixel jitter, and output the filtered pixel coordinates; An image processing host, used to obtain each frame of the main viewing angle image, locate and obtain the target recognition image through pixel coordinates, amplify the target recognition image, and superimpose it on the main viewing angle image in the form of picture-in-picture; The remote touch module is used to receive and display the main viewing angle image after superimposing the target recognition image.

[0012] A storage medium comprises the excavator remote construction tool auxiliary display system as described above.

[0013] An excavator comprises the storage medium as described above.

[0014] The beneficial effects of the technical solution of the present invention are mainly reflected in: 1. This solution obtains the image of the target object in real time while obtaining the main perspective image of the excavator operation, and processes the target recognition image before enlarging it to avoid the target recognition image from being reduced in image accuracy or interfering with the operator due to pixel jitter. At the same time, the enlarged target recognition image is superimposed on the main perspective image in the form of picture-in-picture, which is convenient for observing the surrounding environment and operation details at the same time.

[0015] 2. This solution processes the target recognition image in real time before transmitting the main perspective image to the remote operation terminal. It first compares the position data of the pixel coordinates, and then locates the target recognition image at the recognition position according to the filtered position data. The processing method of the position data is simpler and more efficient. During the processing, invalid data is discarded in time and data is output quickly, further improving the data output efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a method for auxiliary display of remote construction tools for excavators; Figure 2 It is the workflow diagram of the filtering module; Figure 3 is a schematic diagram of the main view image; Figure 4 A schematic diagram of superimposing the target recognition image onto the primary viewing angle image in the form of a picture-in-picture; Figure 5 It is a schematic diagram of the machine correcting the position data of pixel jitter in a moving state and a stationary state respectively; Figure 6 It is a schematic diagram of the working status of the remote construction tool auxiliary display system of the excavator. DETAILED DESCRIPTION

[0017] In order to make the purpose, advantages and features of the present invention more clearly and in detail, the following preferred embodiments are used for illustration and explanation. The embodiments are only typical examples of the application of the technical solution of the present invention. Any technical solution formed by equivalent replacement or equivalent transformation falls within the scope of protection claimed by the present invention.

[0018] The present invention discloses an auxiliary display method for remote construction tools of an excavator. Figure 3 , Figure 6 As shown, the main perspective image of the excavator is captured in real time during the working process of the excavator; in a preferred embodiment, the "real-time capture of the main perspective image of the excavator" means: a front-view high-definition camera 1 with the same driving perspective as that of the excavator is set directly in front of the excavator cab, and the front high-definition camera 1 is used to capture a wide-angle area in front of the cab, and ensure that the target identification object is always within the shooting visible range during the working process of the excavator.

[0019] like Figure 4 As shown, after shooting the main perspective image, the target object in each frame of the main perspective image is identified, and the pixel coordinates of the target object in each frame are extracted; the target object can be selected according to actual work requirements. In some embodiments, the target object is an excavator tool 3 (such as a bucket or a breaker hammer, etc.), and the target object identification method and the pixel coordinate extraction method adopt the existing technology, which will not be repeated here.

[0020] After acquiring the pixel coordinates of the target object, the pixel coordinates of the target object are compared one by one, the pixel coordinate data are subjected to noise reduction processing according to the continuous change trend of the pixel coordinates, the image frames with pixel jitter are filtered out, and the filtered pixel coordinates are output.

[0021] In some embodiments, the step specifically includes: The pixel coordinates of the target identification object in each frame of the main perspective image are obtained in sequence, and the position data in the pixel coordinates are extracted, wherein the pixel coordinate system generally includes an x-axis coordinate and a y-axis coordinate, and the position data includes the x-axis data and the y-axis data in the pixel coordinates.

[0022] like Figure 2 As shown, the position data of the current pixel coordinates and the previous pixel coordinates are compared one by one, and the comparison results are recorded. The pixel coordinate position data of the target object in the current image is compared with the position data in the previous frame image. If the current position data is greater than the previous position data, it is counted as a positive number, and negative numbers are cleared; if the current position data is less than or equal to the previous position data, it is counted as a negative number, and positive numbers are cleared; Set a counting threshold N. When the number of records with the same comparison result is greater than the counting threshold N, output the current position data. Otherwise, discard the current data and maintain the historical data.

[0023] In a preferred embodiment, a filter is used to selectively allow or prevent the position data of the pixel coordinates from passing through. A positive counter and a negative counter are respectively provided in the filter. If the current position data is greater than the previous position data, the positive counter is incremented by 1 and the negative counter is cleared; if the current position data is less than or equal to the previous position data, the negative counter is incremented by 1 and the positive counter is cleared; wherein the counting threshold N is the level of the filter, and then the filtered new position data is outputted after the positive counter or the negative counter exceeds the counting level; and when neither the positive counter nor the negative counter exceeds the counting level, the historical data is outputted, and the historical data is the data outputted by the filter last time, such as Figure 5 As shown, the red line is the change trend of the position data of the current target object, and the dots are the actually collected position data. The position data with different change trends are discarded and replaced with historical data to avoid pixel jitter problems and ensure that the position data is consistent with the position change trend of the target object.

[0024] After obtaining the position data of the pixel coordinates, the recognition position of the target object in the main perspective image is located frame by frame according to the filtered pixel coordinates, and a target recognition image located at the recognition position is obtained; wherein the position data of the pixel coordinates includes x-axis data and y-axis data, and therefore the x-axis data and y-axis data in the pixel coordinates are filtered separately, and the filtered x-axis coordinates and y-axis coordinates are obtained respectively, and then the recognition position of the target object in the main perspective image is located frame by frame according to the x-axis coordinates and y-axis coordinates, and the image at the recognition position is used as the target recognition image.

[0025] The target recognition image is enlarged and superimposed on the main view image in the form of picture-in-picture, and the main view image and its target recognition image are displayed synchronously; wherein, the image enlargement processing and picture-in-picture superposition both adopt existing technologies and are not elaborated here.

[0026] The present invention also discloses an auxiliary display system for remote construction tools of an excavator, such as Figure 6 As shown, including: A camera module is used to capture the main perspective image of the excavator in real time during the working process of the excavator, wherein the camera module at least includes a front-view high-definition camera 1 arranged in front of the excavator cab with the same driving perspective, and the front-view high-definition camera 1 is used to capture the wide-angle area in front of the cab, and ensure that the target identification object is always within the shooting visual range during the working process of the excavator; A recognition module, used to obtain a target object of the excavator in the main view image and extract the pixel coordinates of the target object. In one embodiment, the recognition module is an image processing controller arranged in the cab of the excavator. The image processing controller is connected to the front view high-definition camera 1 by signal to obtain the main view image, identify the target object in the main view image and extract the pixel coordinates of the target object. A filtering module, comprising a filter, wherein the filter comprises a positive counter and a negative counter, which are used to compare the pixel coordinates of the target identification object frame by frame, perform noise reduction processing on the pixel coordinate data according to the continuous change trend of the pixel coordinates, filter out the image frames with pixel jitter, and output the filtered pixel coordinates; wherein the filter is used to selectively allow or prevent the position data of the pixel coordinates from passing through, and the filter is provided with a positive counter and a negative counter respectively, and if the current position data is greater than the previous position data, the positive counter is incremented by 1, and the negative counter is cleared; if the current position data is less than or equal to the previous position data, the negative counter is incremented by 1, and the positive counter is cleared; wherein the counting threshold N is the level of the filter, and then the filtered new position data is output after the positive counter or the negative counter exceeds the counting level; and when neither the positive counter nor the negative counter exceeds the counting level, the historical data is output, and the historical data is the data outputted by the filter last time; The image processing host 2 is used to obtain each frame of the main-view image, locate and obtain the target recognition image through pixel coordinates, amplify the target recognition image, and superimpose it on the main-view image in the form of picture-in-picture. In a preferred embodiment, the image processing host 2 is connected to the filter and the front-view high-definition camera 1 through a 5G network, so as to synchronously obtain the main-view image and the target recognition image; The remote touch module is used to receive and display the main viewing angle image after superimposing the target recognition image, so that the operator can perform operations according to the displayed main viewing angle image and target recognition image.

[0027] The present invention also discloses a storage medium, comprising the excavator remote construction tool auxiliary display system as described above.

[0028] The present invention also discloses an excavator, comprising the storage medium as described above.

[0029] There are many implementation methods of the present invention, and all technical solutions formed by equivalent transformation or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A remote construction tool auxiliary display method for an excavator, characterized in that: The following steps are involved: Real-time capture of the main perspective image of the excavator during its operation; Identify the target object in each frame of the primary view image, and extract the pixel coordinates of the target object in each frame; Comparing the pixel coordinates of the target identification object one by one, performing noise reduction processing on the pixel coordinate data according to the continuous change trend of the pixel coordinates, filtering out the image frames with pixel jitter, and outputting the filtered pixel coordinates; Locate the recognition position of the target object in the primary view image frame by frame according to the filtered pixel coordinates, and obtain the target recognition image located at the recognition position; The target recognition image is enlarged and superimposed on the main viewing angle image in the form of picture-in-picture, and the main viewing angle image and its target recognition image are displayed synchronously.

2. The excavator remote construction tool auxiliary display method according to claim 1 is characterized in that: The "comparing the pixel coordinates of the target identification object one by one, performing noise reduction processing on the pixel coordinate data according to the continuous change trend of the pixel coordinates, filtering out the image frames with pixel jitter, and outputting the filtered pixel coordinates" means: Sequentially obtain the pixel coordinates of the target object in each frame of the main view image, and extract the position data in the pixel coordinates; Compare the position data of the current pixel coordinate with the previous pixel coordinate one by one, and record the comparison results. If the current position data is greater than the previous position data, it is counted as a positive number, and negative numbers are cleared to zero; If the current position data is less than or equal to the previous position data, it will be counted as a negative number, and the positive number will be reset to zero; Set a counting threshold N. When the number of records with the same comparison result is greater than the counting threshold N, output the current position data. Otherwise, discard the current data and maintain the historical data.

3. The excavator remote construction tool auxiliary display method according to claim 2, characterized in that: The position data of the pixel coordinates include x-axis data and y-axis data. The filtered position data are obtained respectively, and the recognition position of the target object in the main perspective image is located frame by frame through each set of position data in turn, and the image of the recognition position is used as the target recognition image.

4. The excavator remote construction tool auxiliary display method according to claim 2, characterized in that: The counting threshold N is the level of the filter.

5. The excavator remote construction tool auxiliary display method according to claim 1, characterized in that: The "real-time shooting of the excavator's main perspective image" means: setting a front-facing high-definition camera with the same driving perspective directly in front of the excavator's cab, using the front-facing high-definition camera to shoot a wide-angle area in front of the cab, and ensuring that the target identification object is always within the shooting visual range.

6. The excavator remote construction tool auxiliary display method according to claim 1, characterized in that: The target identification object is an excavator tool.

7. Excavator remote construction tool auxiliary display system, characterized by: include: A camera module is used to capture the main perspective image of the excavator in real time during the working process of the excavator; A recognition module, used to obtain a target object of the excavator in the primary view image and extract pixel coordinates of the target object; A filtering module, comprising a filter, wherein the filter comprises a positive counter and a negative counter, and is used to compare the pixel coordinates of the target identification object frame by frame, perform noise reduction processing on the pixel coordinate data according to the continuous change trend of the pixel coordinates, filter out the image frames with pixel jitter, and output the filtered pixel coordinates; An image processing host, used to obtain each frame of the main-view image, locate and obtain the target recognition image through pixel coordinates, amplify the target recognition image, and superimpose it on the main-view image in the form of a picture-in-picture; The remote touch module is used to receive and display the main viewing angle image after superimposing the target recognition image.

8. A storage medium, characterized in that: It includes the excavator remote construction tool auxiliary display system as described in claim 7.

9. An excavator, characterized in that: Comprising the storage medium as claimed in claim 8.

Images