A video display method, device and automated guided vehicle

Abstract

Embodiments of the present application provide a video display method, apparatus, and an automated guided vehicle, relating to the technical field of mobile robots. The method includes: displaying a superimposed layer on the display screen of the automated guided vehicle, where the superimposed layer includes a user layer located at the top layer and a video layer located at the bottom layer, a transparent area with a preset size is set at a preset first position of the user layer, and at least one first control corresponding to at least one image acquisition device is set at a preset second position of the user layer; in response to an operation on any one of the first controls, obtaining a video captured by the image acquisition device corresponding to the first control targeted by the operation; scaling the obtained video to a preset size to obtain a target video; displaying the target video at a first position of the video layer, and displaying, through the display screen, a calculation result calculated by the automated guided vehicle based on the obtained video. By using the embodiments of the present application, the solving efficiency of fault problems at the operation site of the automated guided vehicle can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of mobile robots, and particularly to a video display method, apparatus, and automated guided vehicle. Background Art

[0002] An AGV (Automated Guided Vehicle) is an industrial vehicle that loads goods automatically or manually, travels automatically along a set route or tow a load-carrying trolley to a designated location, and then loads and unloads goods automatically or manually. Multiple image acquisition devices are usually arranged on the automated guided vehicle for auxiliary positioning and recognition, such as binocular cameras and code-reading cameras. The binocular camera is used for object recognition, visual navigation, and obstacle avoidance, and the code-reading camera is used for recognizing and reading information such as barcodes and two-dimensional codes to achieve goods management, positioning, and navigation of the automated guided vehicle.

[0003] During the actual operation of the automated guided vehicle, various abnormalities or faults often occur, such as code reading failure, low cargo carrying accuracy, low positioning accuracy, and abnormal binocular obstacle encounter. In order to analyze the above abnormalities or faults, in the related art, the host computer is usually used to obtain the image information collected by the binocular camera and the code-reading camera and perform analysis to determine the fault solution method.

[0004] However, this method depends on the host computer, resulting in low efficiency in solving the fault problems at the operation site of the automated guided vehicle. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a video display method, apparatus, and automated guided vehicle to improve the efficiency of solving fault problems at the operation site of the automated guided vehicle. The specific technical solutions are as follows:

[0006] In the first aspect of the present application, a video display method is provided, which is applied to an automated guided vehicle. The method includes:

[0007] Display a superimposed layer on the display screen of the automated guided vehicle, where the superimposed layer includes a user layer located at the top layer and a video layer located at the bottom layer. A transparent area with a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle;

[0008] In response to an operation on any one of the first controls, obtain the video captured by the image acquisition device corresponding to the first control targeted by the operation;

[0009] Scale the obtained video to the preset size to obtain a target video;

[0010] Display the target video at the first position of the video layer, and display, through the display screen, the calculation result obtained by the automatic guided vehicle based on the acquired video.

[0011] In a possible implementation manner, the image acquisition device of the automatic guided vehicle includes an upper code reading camera;

[0012] The displaying, through the display screen, the calculation result obtained by the automatic guided vehicle based on the acquired video includes:

[0013] If the first control targeted by the operation is the first control corresponding to the upper code reading camera, then display, through the display screen, the pose of the cargo code captured by the upper code reading camera relative to the automatic guided vehicle and the code reading result of the cargo code.

[0014] In a possible implementation manner, the image acquisition device of the automatic guided vehicle includes a lower code reading camera;

[0015] The displaying, through the display screen, the calculation result obtained by the automatic guided vehicle based on the acquired video includes:

[0016] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is QR code navigation, then display, through the display screen, the pose of the ground code captured by the lower code reading camera relative to the automatic guided vehicle, the code reading result of the ground code, and the pose of the automatic guided vehicle in its own operation map.

[0017] In a possible implementation manner, the image acquisition device of the automatic guided vehicle includes a lower code reading camera;

[0018] The displaying, through the display screen, the calculation result obtained by the automatic guided vehicle based on the acquired video includes:

[0019] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is texture navigation, then display, through the display screen, the pose of the automatic guided vehicle in its own operation map calculated based on the ground texture captured by the lower code reading camera.

[0020] In a possible implementation manner, the image acquisition device of the automatic guided vehicle includes a depth camera;

[0021] The displaying the target video at the first position of the video layer and displaying, through the display screen, the calculation result obtained by the automatic guided vehicle based on the acquired video includes:

[0022] Display a second control in the user layer;

[0023] In response to an operation on the second control, display the target video at the first position of the video layer, or display a calculation result obtained by calculating based on the image captured by the depth camera at the first position of the video layer.

[0024] In a possible implementation manner, the depth camera is a binocular camera, and the second control includes a left-eye control, a right-eye control, and a color-map control;

[0025] The displaying the target video at the first position of the video layer in response to an operation on the second control includes:

[0026] In response to an operation on the left-eye control, display the video captured by the left eye of the binocular camera with the preset size at the first position of the video layer;

[0027] In response to an operation on the right-eye control, display the video captured by the right eye of the binocular camera with the preset size at the first position of the video layer;

[0028] In response to an operation on the color-map control, display the color-map video captured by the binocular camera with the preset size at the first position of the video layer.

[0029] In a possible implementation manner, the second control includes an obstacle-map control and a depth-map control;

[0030] The displaying the target video at the first position of the video layer in response to an operation on the second control includes:

[0031] In response to an operation on the obstacle-map control, display an obstacle map obtained by calculating based on the video captured by the depth camera at the first position of the video layer, where the detected obstacles located within a preset obstacle encounter range of the AGV are marked on the obstacle map;

[0032] In response to an operation on the depth-map control, display a depth map obtained by calculating based on the video captured by the depth camera at the first position of the video layer, where all the detected obstacles are marked on the depth map;

[0033] In a possible implementation manner, the method further includes:

[0034] In response to an operation on the second control, display, through the display screen, annotation information indicating whether the automatic guided vehicle encounters an obstacle.

[0035] In a second aspect of the present application, an automated guided vehicle is provided. The automated guided vehicle includes a processor, a display screen, and at least one image acquisition device;

[0036] The image acquisition device is configured to capture and send a video to the processor;

[0037] The processor is configured to, in response to an operation on any one of the first controls, obtain the video captured by the image acquisition device corresponding to the first control targeted by the operation; scale the obtained video to the preset size to obtain a target video; and send the target video and the calculation result calculated based on the obtained video to the display screen;

[0038] The display screen is configured to display an overlay layer, where the overlay layer includes a user layer at the top layer and a video layer at the bottom layer. A transparent area with a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle; display the target video at the first position of the video layer; and display the calculation result.

[0039] In a third aspect of the present application, a video display device is provided, which is applied to an automated guided vehicle. The device includes:

[0040] A layer display module, configured to display an overlay layer on the display screen of the automated guided vehicle, where the overlay layer includes a user layer at the top layer and a video layer at the bottom layer. A transparent area with a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automated guided vehicle;

[0041] A video acquisition module, configured to, in response to an operation on any one of the first controls, obtain the video captured by the image acquisition device corresponding to the first control targeted by the operation;

[0042] A video scaling module, configured to scale the obtained video to the preset size to obtain a target video;

[0043] A video display module, configured to display the target video at the first position of the video layer and display, through the display screen, the calculation result calculated by the automated guided vehicle based on the obtained video.

[0044] In a possible implementation manner, the image acquisition device of the automated guided vehicle includes an upper code reading camera;

[0045] Displaying, on the display screen, the calculation result obtained by the automated guided vehicle based on the acquired video, includes:

[0046] If the first control targeted by the operation is the first control corresponding to the upper code reading camera, display, on the display screen, the pose of the cargo code captured by the upper code reading camera relative to the automated guided vehicle and the code reading result of the cargo code.

[0047] In a possible implementation, the image acquisition device of the automated guided vehicle includes a lower code reading camera;

[0048] The video display module is specifically configured to:

[0049] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automated guided vehicle is QR code navigation, display, on the display screen, the pose of the ground code captured by the lower code reading camera relative to the automated guided vehicle, the code reading result of the ground code, and the pose of the automated guided vehicle in its own operation map.

[0050] In a possible implementation, the image acquisition device of the automated guided vehicle includes a lower code reading camera;

[0051] The video display module is specifically configured to:

[0052] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automated guided vehicle is texture navigation, display, on the display screen, the pose of the automated guided vehicle in its own operation map calculated based on the ground texture captured by the lower code reading camera.

[0053] In a possible implementation, the image acquisition device of the automated guided vehicle includes a depth camera;

[0054] The video display module is specifically configured to:

[0055] Display a second control in the user layer;

[0056] In response to an operation on the second control, display the target video at the first position in the video layer, or display the calculation result calculated based on the image captured by the depth camera at the first position in the video layer.

[0057] In a possible implementation, the depth camera is a binocular camera, and the second control includes a left eye control, a right eye control, and a color map control;

[0058] The displaying the target video at the first position in the video layer in response to an operation on the second control includes:

[0059] In response to an operation on the left-eye control, display, at the first position of the video layer, the video captured by the left eye of the binocular camera with the preset size.

[0060] In response to an operation on the right-eye control, display, at the first position of the video layer, the video captured by the right eye of the binocular camera with the preset size.

[0061] In response to an operation on the color map control, display, at the first position of the video layer, the color map video captured by the binocular camera with the preset size.

[0062] In a possible implementation manner, the second control includes an obstacle map control and a depth map control;

[0063] The "display the target video at the first position of the video layer in response to an operation on the second control" includes:

[0064] In response to an operation on the obstacle map control, display, at the first position of the video layer, an obstacle map calculated based on the video captured by the depth camera, where the detected obstacles located within the preset obstacle encounter range of the automatic guided vehicle are marked on the obstacle map;

[0065] In response to an operation on the depth map control, display, at the first position of the video layer, a depth map calculated based on the video captured by the depth camera, where all detected obstacles are marked on the depth map.

[0066] In a possible implementation manner, the device further includes:

[0067] An information display module, configured to display, through the display screen, annotation information indicating whether the automatic guided vehicle encounters an obstacle in response to an operation on the second control.

[0068] In a fourth aspect of the present application, there is also provided an electronic device, including:

[0069] A memory for storing a computer program;

[0070] A processor, configured to implement the video display method described in any one of the above when executing the program stored in the memory.

[0071] In a fifth aspect of the present application, there is also provided a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the video display method described in any one of the above is implemented.

[0072] An embodiment of the present application also provides a computer program product containing instructions, which when running on a computer, causes the computer to execute the video display method described in any one of the above.

[0073] Beneficial effects of the embodiments of the present application:

[0074] A video display method, device and automatic guided vehicle provided by an embodiment of the present application display a superimposed layer on the display screen of the automatic guided vehicle. The superimposed layer includes a user layer located at the top layer and a video layer located at the bottom layer. A transparent area with a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automatic guided vehicle. The automatic guided vehicle responds to an operation on any one of the first controls, obtains the video captured by the at least one image acquisition device corresponding to the control, scales the obtained video to a preset size to obtain a target video, and finally displays the target video at the first position of the video layer, and displays the result calculated by the automatic guided vehicle based on the obtained video through the display screen. Through the method of the embodiment of the present application, since the superimposed layer includes a user layer located at the top layer and a video layer located at the bottom layer, and a transparent area with a preset size is set at a preset first position of the user layer, and at least one first control corresponding to at least one image acquisition device is set at the preset first position. Therefore, the user can view the target video displayed at the first position of the video layer through the transparent area at the first position on the user layer in the superimposed layer, and the calculation result calculated by the automatic guided vehicle based on the obtained video will also be displayed on the display screen. Compared with the prior art in which the host computer is mainly connected to the automatic guided vehicle first to obtain the video captured by the image acquisition device of the automatic guided vehicle, and then the video displayed by the host computer is combined to analyze the cause of the fault. In the present application, the user can view the calculation result calculated by the automatic guided vehicle based on the video captured by the image acquisition device while viewing the videos captured by each image acquisition device through the display screen of the automatic guided vehicle. Without the participation of the host computer, the cause of the abnormality or fault can be analyzed directly according to the video and calculation result displayed on the display screen of the automatic guided vehicle, which improves the efficiency of solving the fault problem at the operation site of the automatic guided vehicle.

[0075] Of course, implementing any product or method of the present application does not necessarily require achieving all the above advantages at the same time. Description of the Drawings

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other embodiments can also be obtained based on these drawings.

[0077] Figure 1 It is an example diagram for displaying the image information of the code reading camera in the host computer;

[0078] Figure 2 It is an example diagram for displaying the image information of the binocular camera in the host computer;

[0079] Figure 3 It is a schematic diagram of the video display method provided by the embodiment of the present application;

[0080] Figure 4 It is a schematic diagram of the superimposed layer provided by the embodiment of the present application;

[0081] Figure 5a It is the first interface display example diagram of the AGV display screen provided by the embodiment of the present application;

[0082] Figure 5b It is the second interface display example diagram of the AGV display screen provided by the embodiment of the present application;

[0083] Figure 5c It is the third interface display example diagram of the AGV display screen provided by the embodiment of the present application;

[0084] Figure 5d It is the fourth interface display example diagram of the AGV display screen provided by the embodiment of the present application;

[0085] Figure 5e It is the fifth interface display example diagram of the AGV display screen provided by the embodiment of the present application;

[0086] Figure 6 It is an example diagram for displaying the recognition result of the upper code reading camera provided by the embodiment of the present application;

[0087] Figure 7 It is the first example diagram for displaying the recognition result of the lower code reading camera provided by the embodiment of the present application;

[0088] Figure 8 It is the second example diagram for displaying the recognition result of the lower code reading camera provided by the embodiment of the present application;

[0089] Figure 9a It is the first example diagram for displaying the recognition result of the depth camera provided by the embodiment of the present application;

[0090] Figure 9b It is the second example diagram for displaying the recognition result of the depth camera provided by the embodiment of the present application;

[0091] Figure 9c It is the third example diagram for displaying the recognition result of the depth camera provided by the embodiment of the present application;

[0092] Figure 9dThe fourth example diagram for displaying the recognition results of the depth camera provided by the embodiments of the present application;

[0093] Figure 9e The fifth example diagram for displaying the recognition results of the depth camera provided by the embodiments of the present application;

[0094] Figure 9f The sixth example diagram for displaying the recognition results of the depth camera provided by the embodiments of the present application;

[0095] Figure 10 The basic framework diagram of the display software of the display screen of the automatic guided vehicle provided by the embodiments of the present application;

[0096] Figure 11 The structural schematic diagram of the video display device provided by the embodiments of the present application;

[0097] Figure 12 The structural schematic diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0098] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application belong to the scope of protection of the present application.

[0099] First, the professional terms in the embodiments of the present application will be explained:

[0100] Depth camera: A device that can capture the distance information between an object and the camera, including a structured light depth camera and a binocular camera. The structured light depth camera emits light through the camera, and then receives and processes the time difference or phase difference of the reflected light, thereby calculating the distance information and generating a depth image; the binocular camera is an imaging device that mimics the principle of human binocular vision, consisting of two cameras. Two images of the measured object are obtained from different positions by the two cameras, and then the position deviation between the corresponding points in the two images is calculated to obtain the three-dimensional geometric information of the object. The two cameras of the binocular camera are respectively called the left eye and the right eye. Setting a binocular camera on an AGV is usually used for visual navigation or obstacle avoidance.

[0101] Code reading camera: Used to identify textures or two-dimensional codes for texture positioning (texture navigation) or two-dimensional code positioning (two-dimensional code navigation). Usually, two code reading cameras are set on an AGV, one of which is used to identify floor codes or textures, and the other is used to identify cargo codes.

[0102] During the process of AGV carrying out cargo handling operations, the three-dimensional geometric information of each object in the image is calculated using the differences between the images captured by the left and right eyes of the binocular cameras set on the AGV, and then navigation and positioning, obstacle detection and avoidance are realized; One code-reading camera set on the AGV can obtain position information by scanning a preset two-dimensional code or bar code, so as to guide the AGV to drive along a predetermined route. At the same time, another code-reading camera can scan the bar code or two-dimensional code preset on the cargo to obtain information about the cargo, such as type, quantity, destination, etc. However, during the actual operation of the AGV, faults such as unrecognized code reading, low cargo-carrying accuracy, low positioning accuracy, and abnormal binocular obstacle encounter may occur.

[0103] After the above-mentioned faults occur, manual intervention is required for analysis and troubleshooting. In the prior art, usually the host computer is connected to the AGV, and the image information collected by each camera on the AGV is viewed through the host computer. And, for the host computer, usually a frame of the video is intercepted and displayed for analysis. For example, when there is a problem of incorrect code reading recognition at the AGV operation site, in the prior art, the host computer tool is connected to the AGV, and a frame of the video captured by the code-reading camera of the AGV is intercepted through the host computer and viewed in the host computer.

[0104] Although the camera type can be selected as a code-reading camera or a binocular camera in the host computer, for the code-reading camera, only the ground code collected by the code-reading camera, such as a two-dimensional code image, can be displayed on the host computer. For users, they need to analyze according to the code-reading result of the two-dimensional code. However, in the prior art, information such as the code value of the two-dimensional code cannot be displayed in the host computer. As Figure 1 Shown is an example diagram of the image information of the code-reading camera displayed in the host computer. Assuming that there is a problem of incorrect code reading recognition at the AGV operation site, only the two-dimensional code image collected by the code-reading camera is displayed in the host computer, and information such as the code value of the two-dimensional code and the pose of the two-dimensional code relative to the AGV cannot be displayed. Users cannot judge whether the incorrect code reading recognition is caused by the pose of the two-dimensional code relative to the AGV; For the binocular camera, as Figure 2 Shown is an example diagram of the image information of the binocular camera displayed in the host computer. Only the currently captured picture is displayed in the host computer, and information such as obstacle information determined based on the image cannot be displayed. And the above problems will all affect the solution efficiency of the fault problems at the AGV operation site by relevant personnel.

[0105] In order to improve the solution efficiency of the fault problems at the AGV operation site, in the first aspect of the embodiments of the present application, a video display method is provided, which is applied to the AGV, as Figure 3 Shown is the first schematic diagram of the video display method provided by the embodiments of the present application. The method includes the following steps:

[0106] Step S10, display an overlay layer on the display screen of the AGV;

[0107] Among them, the overlay layer includes a user layer at the top layer and a video layer at the bottom layer. A transparent area with a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automatic guided vehicle;

[0108] Step S20, in response to an operation on any one of the first controls, obtain the video captured by the image acquisition device corresponding to the first control targeted by the operation;

[0109] Step S30, scale the obtained video to a preset size to obtain a target video;

[0110] Step S40, display the target video at the first position of the video layer, and display the calculation result calculated by the AGV based on the obtained video through the display screen.

[0111] By using the method of the embodiment of the present application, since the overlay layer includes a user layer at the top layer and a video layer at the bottom layer, and a transparent area with a preset size is set at a preset first position of the user layer, and at least one first control corresponding to at least one image acquisition device is set at the preset first position. Therefore, the user can select the image acquisition device through the first control, and view the target video displayed at the first position of the video layer through the transparent area at the first position on the user layer in the overlay layer. Moreover, the calculation result calculated by the AGV based on the obtained video will also be displayed on the display screen. Compared with the prior art in which the host computer is mainly connected to the AGV first to obtain the video captured by the image acquisition device of the AGV, and then the video displayed by the host computer is combined to analyze the cause of the fault, in the present application, the user can view the calculation result calculated by the AGV based on the video captured by the image acquisition device while viewing the videos captured by each image acquisition device through the display screen of the AGV, without the participation of the host computer. The cause of the abnormality or fault can be directly analyzed according to the video and calculation result displayed on the display screen of the AGV, which improves the efficiency of solving the fault problem at the AGV operation site.

[0112] In addition, during the later maintenance of the AGV, it is also possible to quickly and conveniently conduct problem troubleshooting, reducing the maintenance difficulty and workload.

[0113] Taking the problem of incorrect code reading and recognition occurring at the aforementioned AGV operation site as an example, assume that a code reading camera and a binocular camera are installed on the AGV in this application. Since in the embodiments of this application, a superimposed layer with a user layer as the top layer and a video layer as the bottom layer will be displayed on the display screen of the AGV, and a transparent area with a preset size is set at a preset position of the user layer. After relevant personnel operate the first control for the code reading camera on the display screen of the AGV (hereinafter referred to as the display screen), the AGV will first obtain the video captured by the code reading camera, scale it to a preset size, such as 400×200, and then send it to the preset position of the video layer of the superimposed layer for display. In addition, the AGV will also display the result of the code reading camera reading the two-dimensional code (such as the code value) on the display screen. In this way, relevant personnel can view the video captured by the code reading camera and the code value of the two-dimensional code on the display screen, and analyze the reason for the incorrect code reading of the code reading camera in combination with the video and the two-dimensional code and perform corresponding processing.

[0114] The above steps S10 to S40 will be described in detail below:

[0115] In the above step S10, the user layer refers to the layer used to construct user interface elements when designing the display software of the display screen, and the video layer refers to the layer obtained by embedding a video stream in a separate layer. It can be understood that the sizes of the user layer and the video layer are the same and are the same as the size of the display screen. The superimposed layer is obtained by superimposing the user layer and the video layer in advance, as Figure 4 shown in the schematic diagram of the superimposed layer provided by the embodiments of this application. The user layer is located at the top layer, and the video layer is located at the bottom layer. A transparent area with a preset size is set on the user layer. Since the transparent area is transparent, after the user layer and the video layer are superimposed to obtain the superimposed layer and the video stream is embedded in the video layer, the user can view the video of the video layer through the transparent area on the video layer. Therefore, the transparent area set on the user layer is actually a video preview area for viewing the video. In order to implement the interactive function of the user interface through the user layer, such as the user viewing the videos captured by different image acquisition devices through the user interface, a first control is set at a preset second position on the user layer.

[0116] Among them, the shape of the transparent area is usually a regular shape, such as a rectangle or a circle. The preset first position is used to represent an area, and the preset second position can represent an area or multiple areas. It can be understood that the area represented by the preset first position does not overlap with the area represented by the preset second position.

[0117] The sizes of the areas represented by the preset first position and the preset second position are preset sizes, and the sizes of the areas represented by the preset first position and the preset second position are different. To facilitate the user to view the video, the size of the area represented by the preset first position is larger than the size of the area represented by the preset second position.

[0118] The preset first position and the preset second position can be represented by multiple coordinate points, or can be represented by a coordinate point and a radius. Exemplarily, if A(10, 10), B(10, 60), C(60, 10), D(60, 60) are used to represent the preset first position, then the area represented by the preset first position is a rectangle with a size of 50×50, and then a transparent area with a preset size of 50×50 is generated on the user layer; in another example, if E(20, 20), r = 10 are used to represent the preset first position, then the area represented by the preset first position is a circle with the center at (20, 20) and a radius of 10, and then a transparent area with a preset size of r = 10 is generated on the user layer with the coordinate (20, 20) as the center; if P(1, 1), B(1, 6), C(6, 1), D(6, 6) are used to represent the preset second position, then the area represented by the preset second position is a rectangle with a size of 5×5, and a first control is set in this rectangular area.

[0119] It can be understood that the transparent area on the user layer can be obtained by processing with existing image processing software, or can be obtained by setting the transparency of the preset first position on the user layer, or can be obtained by using other image processing methods. The embodiments of the present application do not limit this.

[0120] The image acquisition device is arranged on the AGV and is used for navigation, obstacle avoidance, cargo recognition and positioning, etc. According to its different functions, the categories of the image acquisition device are also different. For example, a depth camera, an upper code reading camera, a lower code reading camera, etc. The image acquisition device in the embodiments of the present application includes but is not limited to a depth camera and a code reading camera.

[0121] When the preset second position represents an area, a first control is set in this area. The first control can correspond to multiple image acquisition devices or can correspond to one image acquisition device. The image acquisition device corresponding to the first control can be all the image acquisition devices on the AGV, or can be one or part of the image acquisition devices on the AGV.

[0122] Exemplarily, assume that there are 5 image acquisition devices set on the AGV. Then, 5 first controls can be set in the user layer, with each first control corresponding to one image acquisition device. Or, 1 first control can be set in the user layer, and this first control corresponds to 5 image acquisition devices. It can also be that 4 first controls are set in the user layer respectively, where 2 image acquisition devices correspond to the same first control, and the remaining 3 image acquisition devices each correspond to one first control. If the user only has the need to view the images captured by a certain 3 of these 5 image acquisition devices, it can also be that 3 first controls are set in the user layer respectively, and these 3 first controls respectively correspond to these 3 image acquisition devices. It can also be that 1 first control is set in the user layer, and this first control corresponds to these 3 image acquisition devices. It can also be that 2 first controls are set in the user layer respectively, where 2 image acquisition devices correspond to the same first control, and the remaining 1 image acquisition device corresponds to another first control.

[0123] To facilitate the user to accurately select the image acquisition device, each first control corresponds to one image acquisition device. The user can view the videos captured by different image acquisition devices in the video preview area by selecting the first controls corresponding to different image acquisition devices.

[0124] Among them, the first controls can be all displayed on the display screen. Exemplarily, as Figure 5a shown in the first interface display example diagram of the AGV display screen provided by the embodiment of the present application, the first controls corresponding to the image acquisition devices of the AGV are all displayed on the display screen respectively. There is a video preview area 10 on the display interface of the display screen, and different upper code reading camera controls 201, lower code reading camera controls 202, and binocular camera controls 203 are also set on the display screen for the upper code reading camera, the lower code reading camera, and the depth camera respectively.

[0125] The first controls can also be displayed on the display screen through a hidden menu, for example, in the form of a drop-down box. Exemplarily, as Figure 5b shown in the second interface example diagram of the AGV display screen provided by the embodiment of the present application, there is a video preview area 10 and a control 20 for selecting different image acquisition devices on the display interface of the display screen. The user can expand the first controls corresponding to different image acquisition devices through the control 20. As Figure 5c shown, after the user clicks the control 20, the upper code reading camera control 201, the lower code reading camera control 202, and the depth camera control 203 corresponding to the upper code reading camera, the lower code reading camera, and the depth camera will be expanded respectively. It can be understood that the camera names displayed in the control 20 represent the names of the currently selected cameras. For example, Figure 5b in the control 20 in

[0126] If the user selects the upper code reading camera control 201 on the display interface of the display screen, the video collected by the upper code reading camera is displayed in the video preview display area 10. If the user selects the lower code reading camera control 202, the video collected by the lower code reading camera is displayed in the video preview display area 10. If the user clicks the depth camera control 203, the video collected by the depth camera is displayed in the video preview area 10.

[0127] Among them, the depth camera can be a binocular camera, a structured light depth camera, or a depth camera using other technologies. This application does not limit this.

[0128] In order to detect obstacles in all directions of the AGV, different depth cameras are usually set on the AGV for different directions respectively. When the first control is displayed on the display interface in the form of a drop-down box, after the user clicks the control 20, all the first controls corresponding to the image acquisition devices of the AGV can be expanded. As Figure 5d shown, assuming that the image acquisition devices of the AGV include an upper code reading camera, a lower code reading camera, depth camera 1, depth camera 2, and depth camera 3, after the user clicks the control 20, the upper code reading camera control 201, the lower code reading camera control 202, the depth camera 1 control 203, the depth camera 2 control 204, and the depth camera 3 control 205 corresponding to the upper code reading camera, the lower code reading camera, depth camera 1, depth camera 2, and depth camera 3 are respectively displayed.

[0129] In order to avoid excessive controls corresponding to the image acquisition devices from blocking the video screen, in another possible implementation manner, the controls corresponding to the image acquisition devices of the same type can also be displayed through a secondary menu. Exemplarily, as Figure 5e shown, assuming that the image acquisition devices of the AGV include an upper code reading camera, a lower code reading camera, and 5 depth cameras, after the user clicks the control 20, the upper code reading camera control 201 corresponding to the upper code reading camera, the lower code reading camera control 202 corresponding to the lower code reading camera, and a total depth camera control 203 corresponding to all the depth cameras are first displayed. After the user clicks the total depth camera control 203, the depth camera 1 control 2031, the depth camera 2 control 2032, the depth camera 3 control 2033, the depth camera 4 control 2034, and the depth camera 5 control 2035 corresponding to depth camera 1, depth camera 2, depth camera 3, depth camera 4, and depth camera 5 are expanded respectively.

[0130] It can be understood that if there are many controls corresponding to the image acquisition device of the AGV and they cannot all be displayed on the same page, a scroll bar can be used to facilitate the user to view all the image acquisition devices of the AGV. In a possible implementation manner, after the user selects the control corresponding to the image acquisition device, only the number corresponding to the selected image acquisition device is displayed in the area where the control 20 is located.

[0131] Since the sizes of the videos collected by each image acquisition device may not be the same as the size of the video preview area in the overlay layer, if the video collected by the image acquisition device is directly displayed at the first position of the video layer, only a partial picture of the video may be displayed in the video preview area of the overlay layer or the video picture may be too small to be clearly seen, resulting in the user being unable to obtain effective information from the video preview area. Therefore, after the video captured by the image acquisition device corresponding to the first control targeted by the user's operation is obtained in the above step S20, in the above step S30, it is necessary to perform adaptive equal ratio scaling on the obtained video according to the size of the video preview area to ensure that there will be no problem of image distortion caused by scaling when the obtained video is displayed on the display screen of the AGV. Exemplarily, assume that the original video captured by the image acquisition device is obtained in the above step S20, and the size of the original video is 2000×1000, but the preset size of the transparent area set on the user layer in the previous step S10 is 500×500, that is, the size of the video preview area is 500×500. Therefore, the size of the original video can be reduced from 2000×1000 to 1000×500 or 500×250; for another example, assume that the original video captured by the image acquisition device is obtained in the above step S20, and the size of the original video is 500×500, but the preset size of the transparent area set on the user layer in the previous step S10 is 2000×1000, that is, the size of the video preview area is 2000×1000. Therefore, the size of the original video needs to be enlarged from 500×500 to 1000×1000.

[0132] In a possible embodiment, the format of the video collected by the image acquisition device may be different from the format supported by the AGV, or the memory of the video collected by the image acquisition device is large, resulting in a long loading time. In this case, when scaling the obtained video, operations such as rotating, compressing, and format conversion can also be performed on the video so that the target video can be played on the display screen of the AGV, improving the user experience.

[0133] In step S40, displaying the target video at the first position of the video layer means displaying the target video at the first position of the video layer in the superimposed layer, rather than at the first position of the video layer before superimposition. It can be understood that the transparent area of the user layer is directly above the first position of the video layer. After displaying the target video at the first position of the video layer, the target video can be completely displayed in the video preview area of the superimposed layer through the transparent area on the user layer.

[0134] It can be understood that in the embodiment of the present application, each image acquisition device is provided on the AGV. The AGV combines the videos captured by each image acquisition device to calculate the pose of the AGV, the code value of the ground code, the code value of the cargo code, the relative deviation between the AGV and the ground code, the angular position deviation, the obstacle position, etc., so as to realize AGV navigation, obstacle avoidance, cargo recognition and positioning, etc. For the convenience of users to analyze the reasons for anomalies or faults, when the user views the videos captured by each image acquisition device through the display screen of the AGV, the calculation results calculated by the AGV based on the acquired videos will also be displayed on the display screen.

[0135] Exemplarily, if the first control in the foregoing step S20 is a depth camera control, the video obtained in the foregoing step S30 is the video captured by the depth camera, then the AGV can combine the driving route of the AGV and the video captured by the depth camera to calculate the calculation results such as the distances of the objects in the video frame, the object categories, and whether there are obstacles on the driving route; if the first control in the foregoing step S20 is a top code reading control, the video obtained in the foregoing step S30 is the video captured by the top code reading camera, then the AGV can calculate the calculation results based on the video captured by the top code reading camera, such as the pose of the AGV and the relative deviation between the AGV and the ground code.

[0136] It can be seen that according to the different categories of the image devices, the categories of the calculation results are also different. Below, according to the different categories of the image devices, the video display method provided by the embodiment of the present application will be described separately:

[0137] In a possible implementation manner, the image acquisition device of the AGV is a top code reading camera. The top code reading camera is used to read the cargo code. If the first control in the foregoing step S20 is the first control corresponding to the top code reading camera, the calculation results displayed through the display screen are the pose of the cargo code captured by the top code reading camera relative to the AGV and the code reading result of the cargo code.

[0138] The code reading result of the cargo code is the code value of the cargo code. The pose of the cargo code relative to the AGV can be represented by the deviation information of the cargo code relative to the AGV, the turntable angle information, the turntable height information, etc.

[0139] Exemplarily, such as Figure 6The figure shows an example diagram of the recognition result display of the upper code-reading camera provided by the embodiment of the present application. Since the upper code-reading camera is selected, it can be seen that the first control in the above step S20 is the upper code-reading control. In the video preview area 10, a video containing a two-dimensional code image representing the cargo code captured by the upper code-reading camera is displayed. The AGV calculates the code value of the cargo code, the angular position deviation of the cargo code relative to the AGV, the turntable angle (the angle at which the turntable of the AGV rotates relative to the AGV body), and the turntable height (the lifting height of the turntable of the AGV relative to the vehicle body) based on this video. The angular position deviation of 3°, the turntable angle of 90°, the turntable height of 300 mm, and the code-reading result of the cargo code of 10090 are displayed on the display screen. Here, the code-reading result is the code value obtained by recognizing the two-dimensional code in the video captured by the upper code-reading camera.

[0140] By using the method of the embodiment of the present application, the user can not only view the video captured by the upper code-reading camera through the display screen of the AGV, but also view the pose of the cargo code captured by the upper code-reading camera relative to the AGV and the code-reading result of the cargo code. The user can analyze the cause of the abnormality without viewing the host computer, which improves the efficiency of solving the fault problems at the AGV operation site.

[0141] In a possible implementation manner, the image acquisition device of the AGV is a lower code-reading camera. The lower code-reading camera is used for navigation and can be navigated by means of two-dimensional code navigation or texture navigation.

[0142] In the case of navigating by means of two-dimensional code navigation, if the first control in the foregoing step S20 is the first control corresponding to the lower code-reading camera, the calculation results displayed on the display screen are the pose of the AGV relative to the ground code, the code-reading result of the ground code, and the pose of the AGV captured by the lower code-reading camera.

[0143] Exemplarily, as Figure 7 The figure shows the first example diagram of the recognition result display of the lower code-reading camera provided by the embodiment of the present application. Since the lower code-reading control is selected, it can be seen that the first control in the above step S20 is the lower code-reading control. In the video preview area 10, a video containing a two-dimensional code image representing the ground code captured by the lower code-reading camera is displayed. The AGV calculates the code value of the ground code, the pose of the AGV (i.e., the vehicle pose), and the deviation of the AGV relative to the ground code (i.e., the relative deviation) based on this video. The vehicle pose: (12555, 21005), the relative deviation: (5, 4, 0), and the code-reading result of the ground code of 0122502100 are displayed on the display screen.

[0144] When using the method of the embodiment of the present application, when the AGV using QR code navigation has an abnormality or a fault, the user can not only view the video captured by the lower code reading camera through the display screen of the AGV, but also view the pose of the floor code relative to the AGV, the code reading result of the floor code, and the pose of the AGV in its own operation map captured by the lower code reading camera. The user can analyze the cause of the abnormality without viewing the upper computer, which improves the efficiency of solving the fault problem at the AGV operation site.

[0145] In the case of using texture navigation for navigation, if the first control in the foregoing step S20 is the first control corresponding to the lower code reading camera, the calculation result displayed through the display screen is the pose of the AGV in its own operation map.

[0146] Exemplarily, as Figure 8 Shown is the second example diagram of the recognition result display of the lower code reading camera provided by the embodiment of the present application. Since the lower code reading control is selected, it can be seen that the first control in the above step S20 is the lower code reading control. A video containing real-time texture information captured by the lower code reading camera is displayed in the video preview area 10. The AGV calculates its pose (i.e., the trolley pose) based on this video, and the trolley pose: (12555, 21005, 0) is displayed on the display screen.

[0147] When using the method of the embodiment of the present application, when the AGV using texture navigation has an abnormality or a fault, the user can not only view the video captured by the lower code reading camera through the display screen of the AGV, but also view the pose of the AGV, so that the user can determine whether there is an abnormality in the pose of the AGV by comparing the actual pose of the AGV on site with the pose of the AGV on the display screen, without connecting to the upper computer, which improves the efficiency of solving the fault problem at the AGV operation site.

[0148] In a possible implementation manner, the image acquisition device of the AGV is a depth camera. The depth camera is used for obstacle recognition, obstacle avoidance, and positioning. Hereinafter, the binocular camera is taken as an example for illustration. It can be understood that the left eye and the right eye of the binocular camera can respectively collect images, usually RGB images. Based on the position deviation between the corresponding points in the RGB images respectively collected by the left eye and the right eye of the binocular camera, three-dimensional coordinate information can be calculated, and the three-dimensional coordinate information of each point can be represented by a depth map.

[0149] To facilitate the user to select and view the RGB image or the depth image captured by the binocular camera, if the first control in the foregoing step S20 is the first control corresponding to the binocular camera, a second control is displayed in the user layer. When the user clicks the second control, the RGB image captured by the binocular camera or the depth image calculated based on the video captured by the binocular camera is displayed.

[0150] Based on the RGB images collected by the left eye and the right eye of the binocular camera, a stereo image, i.e., a depth image, can also be calculated. Based on the depth image, a corresponding obstacle image can be obtained, and the obstacle image is marked with obstacles and information about the obstacles. Based on this, in a possible implementation manner, the second control includes a left-eye control, a right-eye control, a color-image control, an obstacle-image control, and a depth-image control. Exemplarily, as Figure 9a Shown is the first example diagram of the depth camera recognition result provided by the embodiment of the present application. After the user selects the depth camera through the control 20, there are second controls for different types of images in the display interface, such as the obstacle-image control 21, the depth-image control 22, the color-image control 23, the left-eye control 24, and the right-eye control 25. The user can click on different controls to display the images corresponding to different controls in the video preview area 10 of the display screen, and the recognition result of the depth camera can also be displayed on the display screen. For example, if no obstacle is encountered, "Obstacle state: Binocular normal" will be displayed.

[0151] It can be understood that when the first control is the depth control, the images captured by the depth camera include the images captured by the left eye and the right eye of the binocular camera, and in the above step S30, the images captured by the left eye and the right eye are scaled simultaneously.

[0152] When the user wants to view the video captured by the left eye of the binocular camera, when the user selects the left-eye control 24, the AGV responds to the user's selection of the left-eye control 24 and displays the image captured by the left eye of the binocular camera with a preset size at the first position in the video layer. Exemplarily, as Figure 9b Shown is the second example diagram of the depth camera recognition result provided by the embodiment of the present application. Since the left-eye control is selected, it can be seen that the first control in the above step S20 is the depth control, and the second control in the above text is the left-eye control. The video captured by the left eye of the binocular camera is displayed in the video preview area 10.

[0153] When the user wants to view the video captured by the right eye of the binocular camera, the user selects the right-eye control, and the AGV responds to the user's selection of the right-eye control and displays the video captured by the right eye of the binocular camera with a preset size at the first position in the video layer. Exemplarily, as Figure 9c Shown is the third example diagram of the depth camera recognition result provided by the embodiment of the present application. Since the right-eye control is selected, it can be seen that the first control in the above step S20 is the depth control, and the second control in the above text is the right-eye control. The video captured by the right eye of the binocular camera is displayed in the video preview area 10.

[0154] When the user wants to view the stereoscopic image captured by the binocular camera, the user selects the color image control. In response to the user's selection of the color image control, the AGV displays a stereoscopic video calculated based on the RGB images respectively captured by the left eye and the right eye of the binocular camera at the first position of the video layer, that is, the color image video. Exemplarily, as Figure 9d This is the fourth example diagram showing the recognition result of the depth camera provided by the embodiment of the present application. The color image control is selected. It can be seen that the first control in step S20 above is the depth control, and the second control in the above text is the color image control. A color image video calculated based on the RGB images respectively captured by the left eye and the right eye of the binocular camera is displayed in the video preview area 10.

[0155] By adopting the method of the embodiment of the present application, the user can not only view the stereoscopic image captured by the binocular camera through the display screen of the AGV, but also view the videos respectively captured by the left eye and the right eye of the left eye camera, improving the user experience.

[0156] In a possible implementation manner, after obtaining the two-dimensional image and three-dimensional image captured by the binocular camera, algorithms are used for calculation to obtain a depth map and the position information of each object in the depth map, so as to determine whether there is an obstacle.

[0157] The depth map will display the distance information of all detected surrounding objects relative to the AGV. The distance information is usually represented by gray values. For example, pixels with higher gray values represent objects farther from the AGV, and pixels with lower gray values represent objects closer to the AGV. However, since the visual effect is not obvious when the pixel values of the gray map change slightly, it is difficult to intuitively display the distance information of each object. Therefore, in order to facilitate the user to intuitively know the distance information of each object, in a possible implementation manner, obstacles at different distances in the depth map can be represented by different colors. Obstacles with a distance between the AGV and the preset first distance and the preset second distance are represented by the first color, obstacles with a distance between the AGV and the preset second distance and the preset third distance are represented by the second color, and obstacles with a distance between the AGV and the preset third distance and the preset fourth distance are represented by the third color. It can be understood that each preset distance and each color are set by the user according to requirements, and the embodiment of the present application does not limit this.

[0158] Exemplarily, assume that the user pre-sets that obstacles with a distance of 1 meter to 2 meters from the AGV in the depth map are represented by red, obstacles with a distance of 2 meters to 3 meters from the AGV are represented by yellow, obstacles with a distance of 3 meters to 4 meters from the AGV are represented by blue, and objects with a distance of 4 meters to 5 meters from the AGV are represented by green. The user can view the depth map to understand the distance information of each object and then determine whether there is an obstacle.

[0159] In a possible implementation, the distances between each obstacle in the depth map and the AGV can also be marked in numerical form.

[0160] Although the distance information of all detected obstacles is shown in the depth map, some obstacles do not affect the driving of the AGV. For example, the distance information of the ceiling may be included in the depth image, but the ceiling does not affect the driving of the AGV.

[0161] It can be seen that when the user views the depth map, it is not intuitive to know which obstacles may affect the driving of the AGV. Therefore, in order to facilitate the user to intuitively know the obstacle information, the obstacle map can also be viewed to check whether there are obstacles. Only the detected obstacles within the preset obstacle encounter range of the AGV are marked in the obstacle map. The preset obstacle encounter range is set by the user according to actual experience, and the present application does not limit this.

[0162] Therefore, the second control in the above text includes an obstacle map control and a depth map control. The user can view the obstacle map obtained based on the video captured by the binocular camera by clicking on the obstacle map control. The detected obstacles within the preset obstacle encounter range of the AGV are marked in the obstacle map; click on the depth image control to view the depth map obtained based on the video captured by the binocular camera. The depth map is marked with the depth map calculated based on the video captured by the binocular camera, and all detected obstacles are marked in the depth map.

[0163] When the user wants to view the obstacle map, select the obstacle map control. In response to the user's selection of the obstacle map control, the AGV displays the obstacle map obtained based on the video captured by the binocular camera at the first position of the video layer. Exemplarily, as Figure 9e shown in the fifth example diagram of the depth camera recognition result provided by the embodiment of the present application, the obstacle map control is selected. It can be seen that the first control in step S20 above is the depth control, and the second control in the above text is the obstacle map control. The obstacle map obtained based on the video captured by the binocular camera is displayed in the video preview area 10. The detected obstacles within the preset obstacle encounter range of the AGV are marked in the obstacle map. Obstacles at different distances in the obstacle map can also be identified with different colors. It can be understood that only the obstacles within the preset obstacle encounter range of the AGV are marked in the obstacle map.

[0164] In addition, in order to facilitate the user to know the obstacle closest to the AGV, in a possible implementation, the obstacle closest to the AGV can also be marked with a dotted line box in the obstacle map.

[0165] When the user wants to view the depth image, the depth map control is selected. In response to the user's selection of the depth map control, the AGV displays, at the first position of the video layer, a depth map calculated based on the video captured by the binocular camera, and all detected obstacles are marked in the depth map. Exemplarily, as Figure 9f shown is the sixth example diagram of the display of the depth camera recognition result provided by the embodiment of the present application. The depth map control is selected. It can be seen that the first control in step S20 above is the depth control, and the second control in the above text is the depth map control. A depth map obtained based on the video captured by the binocular camera is displayed in the video preview area 10.

[0166] In order to improve the user experience, in a possible implementation manner, when the user selects the second control, the display screen also displays annotation information indicating whether the AGV encounters an obstacle.

[0167] Exemplarily, as Figure 9f shown, when the user selects the depth map control, the distance of the obstacle marked in the depth map displayed in the video preview area is 1088 mm, and the obstacle is marked with a dashed box. At the same time, "Whether encountering an obstacle: Yes" is used on the display screen to indicate that the AGV encounters an obstacle. It can be understood that when the user selects the left-eye control, the right-eye control, the color map control, or the obstacle map control, the display screen also displays annotation information indicating whether the AGV encounters an obstacle.

[0168] By adopting the method of the embodiment of the present application, an obstacle map is obtained by processing the depth image, and the visual effect of the depth map is improved, so that it is convenient for the user to directly locate the obstacle by combining the depth map and the obstacle map, and the analysis efficiency of anomalies or faults is improved.

[0169] It can be understood that the image displayed in the Figures 6 to 9f video preview area 10 above is only an example. In practice, the video displayed in the video preview area 10 above is the real picture captured by each image acquisition device. The video captured by the binocular camera displayed on the display screen of the AGV can be black and white or color, specifically depending on the configuration of the binocular camera and the user's needs. The embodiment of the present application does not limit this.

[0170] In a possible implementation manner, the user can change the position and size of the transparent area in the user layer through settings, so as to change the size and position of the video preview area of the superimposed layer.

[0171] When the depth camera is a structured light depth camera, since there is no left eye and right eye in the structured light camera, the second control includes a color map control, a depth map control, and an obstacle map control. In this case, the interface diagram of the display screen of the AGV is the same as that in the above Figures 5a to 5eThe same, the example diagram showing the recognition result of the upper code-reading camera, the example diagram showing the recognition result of the lower code-reading camera, and the Figures 6 to 7 The same, the page displayed when the user selects the depth camera control is the same as the Figure 9a Similar, the only difference is that when the depth camera is a structured light depth camera, there are no left-eye and right-eye controls on the page displayed after the user selects the depth camera control. The other example diagrams showing the recognition results of the depth camera are the same as the Figures 9d to 9f The same. It can be understood that when the user selects the color map control, the RGB image collected by the structured light depth camera is displayed on the display screen, that is, the color map video in the previous text.

[0172] To implement the video display method provided in the embodiments of the present application, the basic framework diagram of the display software of the display screen of the AGV provided in the embodiments of the present application is as shown in Figure 10 shown. First, the main process of the application end draws the user layer through the graphical user interface. After the media library receives the image information of the image acquisition device, it rotates, scales, compresses, and converts the format to obtain the video stream of the target video, and then sends the video stream of the target video to the layer module. The layer module embeds the target video onto a new layer to obtain a video layer. After the graphics hardware abstraction layer superimposes the user layer and the video layer, it is sent for display through the driver and multimedia interface of the operating system.

[0173] Among them, the driver of the operating system can be a DMR (Direct Rendering Manager) driver or an FB (Frame buffer) driver. The embodiments of the present application do not limit this.

[0174] Corresponding to the foregoing first aspect, a second aspect of the present application provides an AGV, and the AGV includes a processor, a display screen, and at least one image acquisition device;

[0175] The image acquisition device is used to shoot and send a video to the processor;

[0176] The processor is configured to, in response to an operation on any one of the first controls, obtain the video captured by the image acquisition device corresponding to the first control targeted by the operation; scale the obtained video to the preset size to obtain a target video; and send the target video and the calculation result calculated based on the obtained video to the display screen;

[0177] The display screen is used to display an overlay layer. Among them, the overlay layer includes a user layer located at the top layer and a video layer located at the bottom layer. A transparent area with a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automatic guided vehicle; display the target video at the first position of the video layer; display the calculation result.

[0178] By adopting the embodiment of the present application, since the overlay layer includes a user layer located at the top layer and a video layer located at the bottom layer, and a transparent area with a preset size is set at a preset first position of the user layer, and at least one first control corresponding to at least one image acquisition device is set at the preset first position. Therefore, the user can view the target video displayed at the first position of the video layer through the transparent area at the first position on the user layer in the overlay layer, and the calculation result calculated by the AGV based on the acquired video will also be displayed on the display screen. Compared with the prior art, in which the host computer is mainly connected to the AGV first to obtain the video captured by the image acquisition device of the AGV, and then the video displayed by the host computer is combined to analyze the cause of the fault. In the present application, the user can view the calculation result calculated by the AGV according to the video captured by the image acquisition device while viewing the videos captured by each image acquisition device through the display screen of the AGV. Without the participation of the host computer, the cause of the abnormality or fault can be directly analyzed based on the video and calculation result displayed on the display screen of the AGV, which improves the efficiency of solving the fault problem at the AGV operation site.

[0179] Corresponding to the foregoing first aspect, the third aspect of the present application provides a video display device applied to an AGV, as Figure 11 shown, the device includes:

[0180] A layer display module 1101, configured to display an overlay layer on the display screen of the AGV. Among them, the overlay layer includes a user layer located at the top layer and a video layer located at the bottom layer. A transparent area with a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer. The first control corresponds to at least one image acquisition device of the automatic guided vehicle;

[0181] A video acquisition module 1102, configured to acquire the video captured by the image acquisition device corresponding to the first control targeted by the operation in response to an operation on any one of the first controls;

[0182] A video scaling module 1103, configured to scale the acquired video to the preset size to obtain a target video;

[0183] The video display module 1104 is configured to display the target video at the first position of the video layer and display, through the display screen, the calculation result obtained by the AGV based on the acquired video.

[0184] With the embodiments of the present application, since the superimposed layer includes a user layer at the top layer and a video layer at the bottom layer, and a transparent area with a preset size is set at the preset first position of the user layer, and at least one first control corresponding to at least one image acquisition device is set at the preset first position, therefore, the user can view the target video displayed at the first position of the video layer through the transparent area at the first position on the user layer in the superimposed layer, and moreover, the calculation result obtained by the AGV based on the acquired video will also be displayed on the display screen. Compared with the prior art in which the host computer is mainly connected to the AGV first to obtain the video captured by the image acquisition device of the AGV, and then the video displayed on the host computer is combined to analyze the cause of the failure, in the present application, the user can view, through the display screen of the AGV, the calculation result obtained by the AGV based on the video captured by each image acquisition device while viewing the videos captured by the image acquisition devices, without the participation of the host computer, and directly analyze the cause of the abnormality or failure based on the videos and calculation results displayed on the display screen of the AGV, improving the efficiency of solving the failure problems at the AGV operation site.

[0185] In a possible implementation manner, the image acquisition device of the automatic guided vehicle includes an upper code reading camera;

[0186] The step of displaying, through the display screen, the calculation result obtained by the automatic guided vehicle based on the acquired video includes:

[0187] If the first control targeted by the operation is the first control corresponding to the upper code reading camera, then display, through the display screen, the pose of the cargo code captured by the upper code reading camera relative to the automatic guided vehicle and the code reading result of the cargo code.

[0188] In a possible implementation manner, the image acquisition device of the automatic guided vehicle includes a lower code reading camera;

[0189] The video display module is specifically configured to:

[0190] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is QR code navigation, then display, through the display screen, the pose of the ground code captured by the lower code reading camera relative to the automatic guided vehicle, the code reading result of the ground code, and the pose of the automatic guided vehicle in its own operation map.

[0191] In a possible implementation manner, the image acquisition device of the automatic guided vehicle includes a lower code reading camera;

[0192] The video display module is specifically configured to:

[0193] If the first control targeted by the operation is the first control corresponding to the lower code-reading camera, and the navigation mode of the automatic guided vehicle is texture navigation, then the pose of the automatic guided vehicle in its own operation map calculated based on the ground texture captured by the lower code-reading camera is displayed through the display screen.

[0194] In a possible implementation manner, the image acquisition device of the automatic guided vehicle includes a depth camera;

[0195] The video display module is specifically configured to:

[0196] Display a second control in the user layer;

[0197] In response to an operation on the second control, display the target video at the first position in the video layer, or display a calculation result obtained by calculating based on the image captured by the depth camera at the first position in the video layer.

[0198] In a possible implementation manner, the depth camera is a binocular camera, and the second control includes a left-eye control, a right-eye control, and a color map control;

[0199] The step of displaying the target video at the first position in the video layer in response to an operation on the second control includes:

[0200] In response to an operation on the left-eye control, display the video captured by the left eye of the binocular camera with the preset size at the first position in the video layer;

[0201] In response to an operation on the right-eye control, display the video captured by the right eye of the binocular camera with the preset size at the first position in the video layer;

[0202] In response to an operation on the color map control, display the color map video captured by the binocular camera with the preset size at the first position in the video layer.

[0203] In a possible implementation manner, the second control includes an obstacle map control and a depth map control;

[0204] The step of displaying the target video at the first position in the video layer in response to an operation on the second control includes:

[0205] In response to an operation on the obstacle map control, an obstacle map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and the detected obstacles located within the preset obstacle encounter range of the automated guided vehicle are marked in the obstacle map;

[0206] In response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and all detected obstacles are marked in the depth map.

[0207] In a possible implementation manner, the device further includes:

[0208] An information display module, configured to display, via the display screen, annotation information indicating whether the automated guided vehicle encounters an obstacle in response to an operation on the second control.

[0209] In a fourth aspect of the present application, an electronic device is further provided, as Figure 12 shown, including:

[0210] A memory 1201, configured to store a computer program;

[0211] A processor 1202, configured to implement the video display method described in any one of the above when executing the program stored in the memory.

[0212] The memory may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0213] The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0214] In another embodiment provided by the present application, a computer-readable storage medium is further provided. A computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any of the above video display methods are implemented.

[0215] In another embodiment provided by the present application, a computer program product including instructions is further provided. When it runs on a computer, the computer is enabled to execute any of the video display methods in the above embodiments.

[0216] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, a computer, a server, or a data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrating one or more available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a solid-state disk (SSD), etc.

[0217] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0218] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the automatic guided vehicle and the device, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the relevant parts of the method embodiments for the relevant content.

[0219] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are all included in the protection scope of the present application.

Claims

1. A video display method, characterized in that: Applied to an automatic guided vehicle, the method comprises: Displaying an overlay layer on the display screen of the automatic guided vehicle, wherein the overlay layer includes a user layer at the top layer and a video layer at the bottom layer, a preset first position of the user layer is provided with a transparent area of ​​a preset size, a preset second position of the user layer is provided with at least one first control, the first control corresponds to at least one image acquisition device of the automatic guided vehicle, and an area represented by the preset first position does not overlap with an area represented by the preset second position; In response to an operation on any of the first controls, obtaining a video captured by an image acquisition device corresponding to the first control targeted by the operation; Scaling the acquired video to the preset size to obtain the target video; Displaying the target video at the first position of the video layer, and displaying the calculation result obtained by the automatic guided vehicle based on the acquired video through the display screen; The image acquisition device of the automatic guided vehicle includes an upper code reading camera and a depth camera; The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including: If the first control targeted by the operation is the first control corresponding to the upper barcode reading camera, the position of the cargo code photographed by the upper barcode reading camera relative to the automatic guided vehicle and the barcode reading result of the cargo code are displayed on the display screen, and the position is calculated by the automatic guided vehicle based on the video of the cargo code photographed by the upper barcode reading camera; An obstacle map control and a depth map control are displayed on the user layer; in response to an operation on the obstacle map control, an obstacle map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and detected obstacles within a preset obstacle range of the automatic guided vehicle are marked on the obstacle map; in response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and all detected obstacles are marked on the depth map.

2. The method according to claim 1, characterized in that The image acquisition device of the automatic guided vehicle includes a lower code reading camera; The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including: If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is QR code navigation, the position of the ground code photographed by the lower code reading camera relative to the automatic guided vehicle, the reading result of the ground code, and the position of the automatic guided vehicle in its own operating map are displayed on the display screen.

3. The method according to claim 1, characterized in that The image acquisition device of the automatic guided vehicle includes a lower code reading camera; The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including: If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is texture navigation, the position of the automatic guided vehicle in its own operating map calculated based on the ground texture photographed by the lower code reading camera is displayed through the display screen.

4. The method according to claim 1, characterized in that: The depth camera is a binocular camera; The step of displaying the target video at the first position of the video layer, and displaying the calculation result obtained by the automatic guided vehicle based on the acquired video through the display screen, includes: Display the left eye control, the right eye control and the color picture control in the user layer; In response to an operation on the left-eye control, displaying the video captured by the left eye of the binocular camera of the preset size at the first position of the video layer; In response to an operation on the right-eye control, displaying the video captured by the right eye of the binocular camera of the preset size at the first position of the video layer; In response to an operation on the color image control, the color image video captured by the binocular camera of the preset size is displayed at the first position of the video layer.

5. The method according to claim 1 or 4, characterized in that: The method further comprises: In response to an operation on a depth map control, an obstacle map control, a left eye control, a right eye control, or a color map control, annotation information indicating whether the automatic guided vehicle encounters an obstacle is displayed on the display screen.

6. An automatic guided vehicle, characterized in that: The automatic guided vehicle comprises a processor, a display screen and at least one image acquisition device; the image acquisition device comprises an upper code reading camera; The image acquisition device is used to capture and send video to the processor; The processor is configured to, in response to an operation on any first control, obtain a video captured by an image acquisition device corresponding to the first control targeted by the operation; and scale the obtained video to a preset size to obtain a target video; Sending the target video and a calculation result calculated based on the acquired video to the display screen; The display screen is used to display an overlay layer, wherein the overlay layer includes a user layer at the top layer and a video layer at the bottom layer, a preset first position of the user layer is provided with a transparent area of ​​a preset size, and a preset second position of the user layer is provided with at least one first control, the first control corresponding to at least one image acquisition device of the automatic guided vehicle; displaying the target video at the first position of the video layer; displaying the calculation result; the area represented by the preset first position does not overlap with the area represented by the preset second position; The image acquisition device includes an upper code reading camera and a depth camera; The displaying of the calculation result comprises: Displaying the position of the cargo code captured by the upper code reading camera relative to the automatic guided vehicle and the code reading result of the cargo code, wherein the position is calculated by the automatic guided vehicle based on the video of the cargo code captured by the upper code reading camera; An obstacle map control and a depth map control are displayed on the user layer; in response to an operation on the obstacle map control, an obstacle map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and detected obstacles within a preset obstacle range of the automatic guided vehicle are marked on the obstacle map; in response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and all detected obstacles are marked on the depth map.

7. A video display device, characterized in that: Applied to an automatic guided vehicle, the device comprises: A layer display module, used for displaying an overlay layer on the display screen of the automatic guided vehicle, wherein the overlay layer includes a user layer at the top layer and a video layer at the bottom layer, a preset first position of the user layer is provided with a transparent area of ​​a preset size, a preset second position of the user layer is provided with at least one first control, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle; the area represented by the preset first position does not overlap with the area represented by the preset second position; A video acquisition module, configured to, in response to an operation on any of the first controls, acquire a video captured by an image acquisition device corresponding to the first control targeted by the operation; A video scaling module is used to scale the acquired video to the preset size to obtain a target video; A video display module, configured to display the target video at the first position of the video layer, and to display, through the display screen, a calculation result obtained by the automatic guided vehicle based on the acquired video; The image acquisition device of the automatic guided vehicle includes an upper code reading camera and a depth camera; The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including: If the first control targeted by the operation is the first control corresponding to the upper barcode reading camera, the position of the cargo code photographed by the upper barcode reading camera relative to the automatic guided vehicle and the barcode reading result of the cargo code are displayed on the display screen, and the position is calculated by the automatic guided vehicle based on the video of the cargo code photographed by the upper barcode reading camera; An obstacle map control and a depth map control are displayed on the user layer; in response to an operation on the obstacle map control, an obstacle map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and detected obstacles within a preset obstacle range of the automatic guided vehicle are marked on the obstacle map; in response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and all detected obstacles are marked on the depth map.

8. The device according to claim 7, characterized in that The image acquisition device of the automatic guided vehicle includes a lower code reading camera; The video display module is specifically used for: If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is QR code navigation, the position of the ground code photographed by the lower code reading camera relative to the automatic guided vehicle, the reading result of the ground code, and the position of the automatic guided vehicle in its own running map are displayed on the display screen; and / or The image acquisition device of the automatic guided vehicle includes a lower code reading camera; The video display module is specifically used for: If the first control targeted by the operation is the first control corresponding to the lower barcode reading camera, and the navigation mode of the automatic guided vehicle is texture navigation, the position of the automatic guided vehicle in its own operation map calculated based on the ground texture photographed by the lower barcode reading camera is displayed on the display screen; and / or The depth camera is a binocular camera; The video display module is specifically used for: Displaying a left-eye control, a right-eye control and a color picture control in the user layer; In response to an operation on the left-eye control, displaying the video captured by the left eye of the binocular camera of the preset size at the first position of the video layer; In response to an operation on the right-eye control, displaying the video captured by the right eye of the binocular camera of the preset size at the first position of the video layer; In response to an operation on the color image control, displaying the color image video captured by the binocular camera of the preset size at the first position of the video layer; and / or The device also includes: The information display module is used to respond to the operation of the depth map control, the obstacle map control, the left eye control, the right eye control or the color map control, and display the annotation information indicating whether the automatic guided vehicle encounters an obstacle through the display screen.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-5 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images