Video display method and device and automated guided vehicle

Abstract

The embodiment of the invention provides a video display method and device and an automated guided vehicle, and relates to the technical field of mobile robots. The method comprises the following steps: displaying an overlay layer on a display screen of the automated guided vehicle, the overlay layer comprising a user layer located at a top layer and a video layer located at a bottom layer, and setting a transparent area with a preset size at a preset first position of the user layer, at least one first control corresponding to at least one image acquisition device is arranged at a preset second position of the user layer; in response to the operation for any first control, acquiring a video shot by the image acquisition device corresponding to the first control for which the operation is performed; zooming the obtained video to a preset size to obtain a target video; and displaying the target video at a first position of the video layer, and displaying a calculation result obtained by the automatic guide vehicle based on the obtained video through a display screen. By adopting the embodiment of the invention, the solving efficiency of the fault problem on the operation site of the automated guided vehicle can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of mobile robots, and in particular to a video display method, device and automatic guided vehicle. Background Art

[0002] AGV (Automated Guided Vehicle) is an industrial vehicle that loads goods automatically or manually, drives automatically along a set route or pulls a cargo trolley to a designated location, and then loads and unloads goods automatically or manually. AGVs are usually equipped with multiple image acquisition devices for auxiliary positioning and identification, such as binocular cameras and barcode reading cameras. Binocular cameras are used for object recognition, visual navigation and obstacle avoidance, and barcode reading cameras are used to identify and read barcodes, QR codes and other information to achieve cargo management, positioning and navigation of AGVs.

[0003] In the actual operation of the automated guided vehicle, various anomalies or faults often occur, such as failure to recognize code, low cargo carrying accuracy, low positioning accuracy, and binocular obstacle anomalies. In order to analyze the above anomalies or faults, the relevant technology usually uses a host computer to obtain the image information collected by the binocular camera and the code reading camera and analyze it to determine the fault solution.

[0004] However, this method relies on a host computer, resulting in low efficiency in solving fault problems at the automated guided vehicle operation site. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a video display method, device and automated guided vehicle to improve the efficiency of troubleshooting at the automated guided vehicle operation site. The specific technical solution is as follows:

[0006] In a first aspect of the present application, a video display method is provided, which is applied to an automatic guided vehicle, and the method comprises:

[0007] Displaying an overlay layer on a display screen of the automatic guided vehicle, wherein the overlay layer includes a user layer at a top layer and a video layer at a bottom layer, a transparent area of ​​a preset size is provided at a preset first position of the user layer, at least one first control is provided at a preset second position of the user layer, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle;

[0008] In response to an operation on any of the first controls, obtaining a video captured by an image acquisition device corresponding to the first control targeted by the operation;

[0009] Scaling the acquired video to the preset size to obtain the target video;

[0010] The target video is displayed at the first position of the video layer, and a calculation result obtained by the automatic guided vehicle based on the acquired video is displayed on the display screen.

[0011] In a possible implementation, the image acquisition device of the automatic guided vehicle includes an upper code reading camera;

[0012] The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including:

[0013] If the first control targeted by the operation is the first control corresponding to the upper code reading camera, the position of the cargo code photographed by the upper code reading camera relative to the automatic guided vehicle and the code reading result of the cargo code are displayed on the display screen.

[0014] In a possible implementation, the image acquisition device of the automatic guided vehicle includes a lower code reading camera;

[0015] The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including:

[0016] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is QR code navigation, the position of the ground code photographed by the lower code reading camera relative to the automatic guided vehicle, the reading result of the ground code, and the position of the automatic guided vehicle in its own operating map are displayed on the display screen.

[0017] In a possible implementation, the image acquisition device of the automatic guided vehicle includes a lower code reading camera;

[0018] The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including:

[0019] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is texture navigation, the position of the automatic guided vehicle in its own operating map calculated based on the ground texture photographed by the lower code reading camera is displayed through the display screen.

[0020] In a possible implementation, the image acquisition device of the automatic guided vehicle includes a depth camera;

[0021] The step of displaying the target video at the first position of the video layer, and displaying the calculation result obtained by the automatic guided vehicle based on the acquired video through the display screen, includes:

[0022] Displaying a second control in the user layer;

[0023] In response to an operation on the second control, the target video is displayed at the first position of the video layer, or a calculation result calculated based on the picture captured by the depth camera is displayed at the first position of the video layer.

[0024] In a possible implementation, the depth camera is a binocular camera, and the second control includes a left-eye control, a right-eye control, and a color picture control;

[0025] In response to the operation on the second control, displaying the target video at the first position of the video layer includes:

[0026] In response to an operation on the left-eye control, displaying the video captured by the left eye of the binocular camera of the preset size at the first position of the video layer;

[0027] In response to an operation on the right-eye control, displaying the video captured by the right eye of the binocular camera of the preset size at the first position of the video layer;

[0028] In response to an operation on the color image control, the color image video captured by the binocular camera of the preset size is displayed at the first position of the video layer.

[0029] In a possible implementation manner, the second control includes an obstacle map control and a depth map control;

[0030] In response to the operation on the second control, displaying the target video at the first position of the video layer includes:

[0031] In response to an operation on the obstacle map control, an obstacle map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, wherein the obstacle map is marked with detected obstacles within a preset obstacle range of the AGV;

[0032] In response to an operation on the depth map control, displaying a depth map calculated based on the video captured by the depth camera at the first position of the video layer, wherein all detected obstacles are marked in the depth map;

[0033] In a possible implementation, the method further includes:

[0034] In response to the operation on the second control, labeling information indicating whether the automatic guided vehicle encounters an obstacle is displayed on the display screen.

[0035] In a second aspect of the present application, an automatic guided vehicle is provided, the automatic guided vehicle comprising a processor, a display screen, and at least one image acquisition device;

[0036] The image acquisition device is used to capture and send video to the processor;

[0037] The processor is used for, in response to an operation on any of the first controls, acquiring a video captured by an image acquisition device corresponding to the first control targeted by the operation; scaling the acquired video to the preset size to obtain a target video; and sending the target video and a calculation result calculated based on the acquired video to the display screen;

[0038] The display screen is used to display overlay layers, wherein the overlay layers include a user layer at the top layer and a video layer at the bottom layer, a transparent area of ​​a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle; the target video is displayed at the first position of the video layer; and the calculation result is displayed.

[0039] In a third aspect of the present application, a video display device is provided, which is applied to an automatic guided vehicle, and the device comprises:

[0040] A layer display module, configured to display an overlay layer on a display screen of the automatic guided vehicle, wherein the overlay layer includes a user layer at a top layer and a video layer at a bottom layer, a transparent area of ​​a preset size is provided at a preset first position of the user layer, and at least one first control is provided at a preset second position of the user layer, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle;

[0041] A video acquisition module, configured to, in response to an operation on any of the first controls, acquire a video captured by an image acquisition device corresponding to the first control targeted by the operation;

[0042] A video scaling module is used to scale the acquired video to the preset size to obtain a target video;

[0043] A video display module is used to display the target video at the first position of the video layer, and to display the calculation results obtained by the automatic guided vehicle based on the acquired video through the display screen.

[0044] In a possible implementation, the image acquisition device of the automatic guided vehicle includes an upper code reading camera;

[0045] The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including:

[0046] If the first control targeted by the operation is the first control corresponding to the upper code reading camera, the position of the cargo code photographed by the upper code reading camera relative to the automatic guided vehicle and the code reading result of the cargo code are displayed on the display screen.

[0047] In a possible implementation, the image acquisition device of the automatic guided vehicle includes a lower code reading camera;

[0048] The video display module is specifically used for:

[0049] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is QR code navigation, the position of the ground code photographed by the lower code reading camera relative to the automatic guided vehicle, the reading result of the ground code, and the position of the automatic guided vehicle in its own operating map are displayed on the display screen.

[0050] In a possible implementation, the image acquisition device of the automatic guided vehicle includes a lower code reading camera;

[0051] The video display module is specifically used for:

[0052] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is texture navigation, the position of the automatic guided vehicle in its own operating map calculated based on the ground texture photographed by the lower code reading camera is displayed through the display screen.

[0053] In a possible implementation, the image acquisition device of the automatic guided vehicle includes a depth camera;

[0054] The video display module is specifically used for:

[0055] Displaying a second control in the user layer;

[0056] In response to an operation on the second control, the target video is displayed at the first position of the video layer, or a calculation result calculated based on the picture captured by the depth camera is displayed at the first position of the video layer.

[0057] In a possible implementation, the depth camera is a binocular camera, and the second control includes a left-eye control, a right-eye control, and a color picture control;

[0058] In response to the operation on the second control, displaying the target video at the first position of the video layer includes:

[0059] In response to an operation on the left-eye control, displaying the video captured by the left eye of the binocular camera of the preset size at the first position of the video layer;

[0060] In response to an operation on the right-eye control, displaying the video captured by the right eye of the binocular camera of the preset size at the first position of the video layer;

[0061] In response to an operation on the color image control, the color image video captured by the binocular camera of the preset size is displayed at the first position of the video layer.

[0062] In a possible implementation manner, the second control includes an obstacle map control and a depth map control;

[0063] In response to the operation on the second control, displaying the target video at the first position of the video layer includes:

[0064] In response to an operation on the obstacle map control, displaying an obstacle map calculated based on the video captured by the depth camera at the first position of the video layer, wherein the obstacle map is marked with detected obstacles within a preset obstacle range of the automatic guided vehicle;

[0065] In response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and all detected obstacles are marked in the depth map.

[0066] In a possible implementation, the device further includes:

[0067] An information display module is used to display, in response to an operation on the second control, labeling information indicating whether the automatic guided vehicle encounters an obstacle through the display screen.

[0068] In a fourth aspect of the present application, an electronic device is further provided, including:

[0069] Memory, used to store computer programs;

[0070] The processor is used to implement any of the above-mentioned video display methods when executing the program stored in the memory.

[0071] In a fifth aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned video display methods is implemented.

[0072] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned video display methods.

[0073] Beneficial effects of the embodiments of the present application:

[0074] A video display method, device and automatic guided vehicle provided in an embodiment of the present application display an overlay layer on a display screen of the automatic guided vehicle, wherein the overlay layer includes a user layer at the top layer and a video layer at the bottom layer, and a transparent area of ​​a preset size is provided at a preset first position of the user layer, and at least one first control is provided at a preset second position of the user layer, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle. In response to an operation on any first control, the automatic guided vehicle obtains a video captured by at least one image acquisition device corresponding to the control, scales the obtained video to a preset size, obtains a target video, and finally displays the target video at the first position of the video layer, and displays a result calculated by the automatic guided vehicle based on the obtained video through the display screen. Through the method of the embodiment of the present application, since the overlay layer includes a user layer located at the top layer and a video layer located at the bottom layer, and the preset first position of the user layer is provided with a transparent area of ​​a preset size, and the preset first position is provided with at least one first control corresponding to at least one image acquisition device, the user can browse the target video displayed at the first position of the video layer through the transparent area at the first position on the user layer in the overlay layer, and the display screen will also display the calculation results obtained by the automatic guided vehicle based on the acquired video. Compared with the prior art, in which the host computer is mainly connected to the automatic guided vehicle to obtain the video captured by the image acquisition device of the automatic guided vehicle, and then the cause of the fault is analyzed in combination with the video displayed by the host computer, in the present application, the user can view the videos captured by each image acquisition device through the display screen of the automatic guided vehicle while viewing the calculation results obtained by the automatic guided vehicle based on the video captured by the image acquisition device. Without the participation of the host computer, the cause of the abnormality or fault can be analyzed directly based on the video and calculation results displayed on the display screen of the automatic guided vehicle, thereby improving the efficiency of troubleshooting at the operation site of the automatic guided vehicle.

[0075] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0077] Figure 1 This is an example of the image information of the code reading camera displayed in the host computer;

[0078] Figure 2 This is an example diagram showing the image information of the binocular camera in the host computer;

[0079] Figure 3 A schematic diagram of a video display method provided in an embodiment of the present application;

[0080] Figure 4 A schematic diagram of an overlay layer provided in an embodiment of the present application;

[0081] Figure 5a This is an example diagram of the first interface display of the AGV display screen provided in the embodiment of the present application;

[0082] Figure 5b This is an example diagram of the second interface display of the AGV display screen provided in the embodiment of the present application;

[0083] Figure 5c This is a third interface display example diagram of the AGV display screen provided in the embodiment of the present application;

[0084] Figure 5d This is a fourth interface display example diagram of the AGV display screen provided in an embodiment of the present application;

[0085] Figure 5e This is a fifth example diagram of the interface display of the AGV display screen provided in the embodiment of the present application;

[0086] Figure 6 An example diagram showing the recognition results of the upper barcode reading camera provided in the embodiment of the present application;

[0087] Figure 7 The first example diagram showing the recognition result of the lower barcode reading camera provided in the embodiment of the present application;

[0088] Figure 8 A second example diagram showing the recognition result of the lower barcode reading camera provided in the embodiment of the present application;

[0089] Figure 9a A first example diagram showing the depth camera recognition results provided in an embodiment of the present application;

[0090] Figure 9b A second example diagram showing the depth camera recognition results provided in an embodiment of the present application;

[0091] Fig.9c A third example diagram showing the depth camera recognition result provided in an embodiment of the present application;

[0092] Figure 9dA fourth example diagram showing the depth camera recognition result provided in an embodiment of the present application;

[0093] Fig.9e A fifth example diagram showing the depth camera recognition result provided in an embodiment of the present application;

[0094] Figure 9f A sixth example diagram showing the depth camera recognition result provided in an embodiment of the present application;

[0095] Fig.10 A basic framework diagram of the display software of the display screen of the automatic guided vehicle provided in the embodiment of the present application;

[0096] Fig.11 A schematic diagram of the structure of a video display device provided in an embodiment of the present application;

[0097] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0099] First, the professional terms in the embodiments of the present application are explained:

[0100] Depth camera: a device that can capture the distance information between an object and a camera, including structured light depth cameras and binocular cameras. A structured light depth camera receives and processes the time difference or phase difference of the light emitted by the camera, thereby calculating the distance information and generating a depth image; a binocular camera is an imaging device that imitates the principle of human binocular vision. It consists of two cameras, which obtain two images of the object being measured from different positions through the two cameras, and then obtain the three-dimensional geometric information of the object by calculating the position deviation between the corresponding points in the two images. The two cameras of a binocular camera are called the left eye and the right eye, respectively. Setting up a binocular camera on an AGV is usually used for visual navigation or obstacle avoidance.

[0101] Barcode reading camera: used to identify textures or QR codes, and to perform texture positioning (texture navigation) or QR code positioning (QR code navigation). Usually two barcode reading cameras are set up on the AGV, one of which is used to identify the ground code or texture, and the other is used to identify the cargo code.

[0102] During the cargo handling operation of AGV, the difference between the images collected by the left and right eyes of the binocular camera installed on the AGV is used to calculate the three-dimensional geometric information of each object in the image, thereby realizing navigation and positioning, obstacle detection and obstacle avoidance; a code reading camera installed on the AGV can obtain location information by scanning the preset QR code or barcode, thereby guiding the AGV to travel along the predetermined route. At the same time, another code reading camera can scan the preset barcode or QR code on the cargo to obtain cargo information, such as type, quantity, destination, etc. However, during the actual operation of the AGV, there may be faults such as code reading failure, low cargo carrying accuracy, low positioning accuracy, and binocular obstacle abnormalities.

[0103] After the above fault occurs, manual intervention is required for analysis and troubleshooting. In the prior art, the host computer is usually connected to the AGV, and the image information collected by each camera on the AGV is viewed through the host computer. In addition, the host computer usually captures and displays a frame of the video for analysis. For example, if a code reading and recognition error occurs at the AGV operation site, the prior art is to connect the host computer tool to the AGV, and use the host computer to capture a frame of the video captured by the AGV's code reading camera and view the frame in the host computer.

[0104] Although the host computer can select the camera type as a code reading camera or a binocular camera, for a code reading camera, the host computer can only display the ground code collected by the code reading camera, such as a QR code image. For users, it is necessary to analyze the QR code reading result. However, in the prior art, the host computer cannot display the code value and other information of the QR code. Figure 1 The figure shows an example of the image information of the barcode reading camera displayed in the host computer. Assuming that a barcode reading and recognition error occurs at the AGV operation site, the host computer only displays the QR code image collected by the barcode reading camera, and cannot display the QR code value, the position of the QR code relative to the AGV, and other information. The user cannot determine whether the barcode reading and recognition error is caused by the position of the QR code relative to the AGV. For a binocular camera, such as Figure 2 The figure shows an example of the image information of the binocular camera displayed in the host computer. The host computer only displays the currently captured image, and cannot display information such as obstacles determined based on the image. The above problems will affect the efficiency of relevant personnel in solving problems at the AGV operation site.

[0105] In order to improve the efficiency of solving fault problems at the AGV operation site, a first aspect of the embodiment of the present application provides a video display method, which is applied to the AGV, such as Figure 3 The first schematic diagram of the video display method provided in the embodiment of the present application is shown, and the method comprises the following steps:

[0106] Step S10, displaying the overlay layer on the display screen of the AGV;

[0107] The overlay layer includes a user layer at the top layer and a video layer at the bottom layer, a preset first position of the user layer is provided with a transparent area of ​​a preset size, and a preset second position of the user layer is provided with at least one first control, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle;

[0108] Step S20, in response to an operation on any first control, acquiring a video captured by an image acquisition device corresponding to the first control targeted by the operation;

[0109] Step S30, scaling the acquired video to a preset size to obtain a target video;

[0110] Step S40, displaying the target video at the first position of the video layer, and displaying the calculation result obtained by the AGV based on the acquired video through the display screen.

[0111] According to the method of the embodiment of the present application, since the overlay layer includes a user layer at the top layer and a video layer at the bottom layer, and a preset first position of the user layer is provided with a transparent area of ​​a preset size, and the preset first position is provided with at least one first control corresponding to at least one image acquisition device, the user can select the image acquisition device through the first control, and browse the target video displayed at the first position of the video layer through the transparent area at the first position on the user layer in the overlay layer, and the display screen will also display the calculation results calculated by the AGV based on the acquired video. Compared with the prior art, in which the host computer is mainly connected to the AGV to obtain the video captured by the image acquisition device of the AGV, and then the cause of the fault is analyzed in combination with the video displayed by the host computer, in the present application, the user can view the videos captured by each image acquisition device through the display screen of the AGV while viewing the calculation results calculated by the AGV based on the video captured by the image acquisition device. Without the participation of the host computer, the cause of the abnormality or fault can be analyzed directly based on the video and calculation results displayed on the display screen of the AGV, thereby improving the efficiency of troubleshooting at the AGV operation site.

[0112] In addition, when the AGV is maintained in the later stage, problems can be easily and quickly checked, reducing the difficulty and workload of maintenance.

[0113] Still taking the problem of code reading and recognition errors occurring at the aforementioned AGV operation site as an example, assuming that the AGV in the present application is provided with a code reading camera and a binocular camera, since in the embodiment of the present application an overlay layer with a user layer as the top layer and a video layer as the bottom layer will be displayed on the display screen of the AGV, and a transparent area of ​​a preset size is provided at a preset position of the user layer, after the relevant personnel operate the first control for the code reading camera on the display screen (hereinafter referred to as the display screen) of the AGV on site, the AGV will first obtain the video taken by the code reading camera, and scale it to a preset size, such as 400×200, and then send it to the preset position of the video layer of the overlay layer for display, and the AGV will also display the result of the code reading camera reading the QR code (such as the code value) on the display screen, so that the relevant personnel can view the video and the code value of the QR code captured by the code reading camera on the display screen, analyze the cause of the code reading error of the code reading camera in combination with the video and the QR code, and perform corresponding processing.

[0114] The above steps S10 to S40 are described in detail below:

[0115] In the above step S10, the user layer refers to the layer used to construct the user interface elements when designing the display software of the display screen, and the video layer refers to the layer obtained by embedding the video stream in a separate layer. It can be understood that the size of the user layer and the video layer is the same and the same as the size of the display screen. The overlay layer is obtained by overlaying the user layer and the video layer in advance, such as Figure 4 The schematic diagram of the overlay layer provided by the embodiment of the present application is shown, the user layer is located at the top layer, the video layer is located at the bottom layer, and a transparent area of ​​a preset size is set on the user layer. Since the transparent area is transparent, after the user layer and the video layer are superimposed to obtain the overlay layer, after the video stream is embedded on the video layer, the user can view the video of the video layer through the transparent area on the video layer. Therefore, the transparent area set on the user layer is actually a video preview area for viewing the video. In order to realize the interactive function of the user interface through the user layer, such as the user viewing the videos collected by different image acquisition devices through the user interface, a first control is set at the preset second position of the user layer.

[0116] The shape of the transparent area is usually a regular shape, such as a rectangle or a circle. The preset first position is used to represent an area, and the preset second position can represent an area or multiple areas. It can be understood that the area represented by the preset first position does not overlap with the area represented by the preset second position.

[0117] The sizes of the areas represented by the preset first position and the preset second position are preset sizes, and the sizes of the areas represented by the preset first position and the preset second position are different. In order to facilitate users to browse videos, the size of the area represented by the preset first position is larger than the size of the area represented by the preset second position.

[0118] The preset first position and the preset second position can be represented by multiple coordinate points, or by coordinate points and radius. For example, A (10, 10), B (10, 60), C (60, 10), D (60, 60) represent the preset first position, and the area represented by the preset first position is a rectangle with a size of 50×50, and then a transparent area with a preset size of 50×50 is generated on the user layer; in another example, E (20, 20) and r = 10 represent the preset first position, and then the area represented by the preset first position is a circle with a center at (20, 20) and a radius of 10, and then a transparent area with a preset size of r = 10 is generated on the user layer with the coordinate (20, 20) as the center; if P (1, 1), B (1, 6), C (6, 1), D (6, 6) represent the preset second position, then the area represented by the preset second position is a rectangle with a size of 5×5, and the first control is set in the rectangular area.

[0119] It is understandable that the transparent area on the user layer can be obtained by processing through existing image processing software, or by setting the transparency of a preset first position on the user layer, or by using other image processing methods, and the embodiments of the present application are not limited to this.

[0120] The image acquisition device is installed on the AGV and is used for navigation, obstacle avoidance, cargo identification and positioning, etc. The types of image acquisition devices vary according to their functions, such as depth cameras, upper code reading cameras, lower code reading cameras, etc. The image acquisition devices in the embodiments of the present application include but are not limited to depth cameras and code reading cameras.

[0121] When the preset second position represents an area, the area is provided with a first control, and the first control may correspond to multiple image acquisition devices or one image acquisition device. The image acquisition device corresponding to the first control may be all image acquisition devices on the AGV or one or part of the image acquisition devices on the AGV.

[0122] For example, assuming that there are five image acquisition devices on the AGV, five first controls may be set in the user layer, each of which corresponds to one image acquisition device. One first control may be set in the user layer, and the first control corresponds to five image acquisition devices. Four first controls may be set in the user layer, two of which correspond to the same first control, and the remaining three correspond to one first control. If the user only needs to view images taken by three of the five image acquisition devices, three first controls may be set in the user layer, and the three first controls correspond to the three image acquisition devices. One first control may be set in the user layer, and the first control corresponds to the three image acquisition devices. Two first controls may be set in the user layer, two of which correspond to the same first control, and the remaining one corresponds to another first control.

[0123] In order to facilitate the user to accurately select the image acquisition device, each first control corresponds to an image acquisition device, and the user can view the videos acquired by different image acquisition devices in the video preview area by selecting the first controls corresponding to different image acquisition devices.

[0124] The first control may be displayed entirely on the display screen. For example, Figure 5a The figure shows an example diagram of the first interface display of the AGV display screen provided in an embodiment of the present application. The first controls corresponding to the image acquisition device of the AGV are all displayed in the display screen. There is a video preview area 10 on the display interface of the display screen. The display screen is also provided with different upper code reading camera controls 201, lower code reading camera controls 202 and binocular camera controls 203 for the upper code reading camera, lower code reading camera and depth camera, respectively.

[0125] The first control may also be displayed on the display screen through a hidden menu, such as a drop-down box. Figure 5b The second interface example diagram of the AGV display screen provided in the embodiment of the present application is shown. The display interface of the display screen has a video preview area 10 and a control 20 for selecting different image acquisition devices. The user can use the control 20 to expand the first control corresponding to the different image acquisition devices, such as Figure 5c As shown, after the user clicks the control 20, the upper code reading camera control 201, the lower code reading camera control 202 and the depth camera control 203 corresponding to the upper code reading camera, the lower code reading camera and the depth camera will be expanded. It can be understood that the camera name displayed in the control 20 indicates the name of the currently selected camera, such as Figure 5b The display “Depth Camera 1” in the control 20 indicates that the currently selected camera is depth camera 1.

[0126] If the user selects the upper code reading camera control 201 in the display interface of the display screen, the video captured by the upper code reading camera is displayed in the video preview display area 10; if the user selects the lower code reading camera control 202, the video captured by the lower code reading camera is displayed in the video preview display area 10; if the user clicks the depth camera control 203, the video captured by the depth camera is displayed in the video preview area 10.

[0127] Among them, the depth camera can be a binocular camera, a structured light depth camera, or a depth camera using other technologies, which is not limited in this application.

[0128] In order to detect obstacles in all directions of the AGV, different depth cameras are usually set on the AGV for different directions. When the first control is displayed in the display interface in the form of a drop-down box, the user clicks the control 20 to expand the first control corresponding to all image acquisition devices of the AGV. Figure 5d As shown, assuming that the image acquisition device of the AGV includes an upper code reading camera, a lower code reading camera, a depth camera 1, a depth camera 2 and a depth camera 3, after the user clicks the control 20, the upper code reading camera control 201, the lower code reading camera control 202, the depth camera 1 control 203, the depth camera 2 control 204, and the depth camera 3 control 205 corresponding to the upper code reading camera, the lower code reading camera, the depth camera 1, the depth camera 2 and the depth camera 3 will be displayed respectively.

[0129] In order to prevent the controls corresponding to the image acquisition device from blocking the video screen too much, in another possible implementation, the controls corresponding to the image acquisition device of the same type may also be displayed through a secondary menu. Figure 5e As shown, assuming that the image acquisition device of the AGV includes an upper code reading camera, a lower code reading camera and 5 depth cameras, after the user clicks the control 20, the upper code reading camera control 201 corresponding to the upper code reading camera, the lower code reading camera control 202 corresponding to the lower code reading camera and a total depth camera control 203 corresponding to all depth cameras are first displayed. After the user clicks the total depth camera control 203, the depth camera 1 control 2031, the depth camera 2 control 2032, the depth camera 3 control 2033, the depth camera 4 control 2034, and the depth camera 5 control 2035 corresponding to the depth camera 1, depth camera 2, depth camera 3, depth camera 4 and depth camera 5 are expanded respectively.

[0130] It is understandable that if there are many controls corresponding to the image acquisition devices of the AGV and they cannot all be displayed on the same page, a scroll bar can be used to facilitate the user to view all the image acquisition devices of the AGV. In a possible implementation, after the user selects the control corresponding to the image acquisition device, only the number corresponding to the selected image acquisition device is displayed in the area where the control 20 is located.

[0131] Since the size of the video captured by each image acquisition device may not be the same as the size of the video preview area in the overlay layer, if the video captured by the image acquisition device is directly displayed in the first position of the video layer, the video preview area of ​​the overlay layer may only display a partial picture of the video or the video picture is too small to be seen clearly, resulting in the user being unable to obtain effective information from the video preview area. Therefore, after obtaining the video captured by the image acquisition device corresponding to the first control targeted by the user's operation in the above step S20, the obtained video needs to be adaptively scaled proportionally according to the size of the video preview area in the above step S30 to ensure that there will be no image distortion caused by scaling when the obtained video is displayed on the AGV display screen. Exemplarily, assuming that the original video captured by the image acquisition device is obtained in the above step S20, and the size of the original video is 2000×1000, but the preset size of the transparent area set on the user layer in the above step S10 is 500×500, that is, the size of the video preview area is 500×500, so the size of the original video can be reduced from 2000×1000 to 1000×500 or 500×250; for another example, assuming that the original video captured by the image acquisition device is obtained in the above step S20, and the size of the original video is 500×500, but the preset size of the transparent area set on the user layer in the above step S10 is 2000×1000, that is, the size of the video preview area is 2000×1000, so the size of the original video needs to be enlarged from 500×500 to 1000×1000.

[0132] In a possible embodiment, the format of the video captured by the image acquisition device may be different from the format supported by the AGV, or the video memory captured by the image acquisition device is large, resulting in a long loading time. In this case, when scaling the acquired video, the video can also be rotated, compressed, and format converted, so that the target video can be played on the display screen of the AGV, thereby improving the user experience.

[0133] Displaying the target video at the first position of the video layer in step S40 means displaying the target video at the first position of the video layer of the overlay layer, rather than displaying the target video at the first position of the video layer before overlaying. It is understandable that the transparent area of ​​the user layer is directly above the first position of the video layer. After the target video is displayed at the first position of the video layer, the target video can be completely displayed in the video preview area of ​​the overlay layer through the transparent area on the user layer.

[0134] It is understandable that the AGV in the embodiment of the present application is provided with various image acquisition devices, and the AGV calculates the AGV's posture, the code value of the ground code, the code value of the cargo code, the relative deviation between the AGV and the ground code, the angle position deviation, the obstacle position, etc., in combination with the video captured by each image acquisition device, so as to realize AGV navigation, obstacle avoidance, cargo identification and positioning, etc. In order to facilitate the user to analyze the cause of abnormality or failure, when the user views the video captured by each image acquisition device through the display screen of the AGV, the calculation results calculated by the AGV based on the acquired video will also be displayed on the display screen.

[0135] Exemplarily, if the first control in the aforementioned step S20 is a depth camera control, the video obtained in the aforementioned step S30 is a video captured by the depth camera, and the AGV can combine the AGV's driving route and the video taken by the depth camera to calculate the distance of each object in the video screen, the object category, and whether there are obstacles on the driving route, etc.; if the first control in the aforementioned step S20 is an upper code reading control, the video obtained in the aforementioned step S30 is a video captured by the upper code reading camera, and the AGV can calculate the calculation results based on the video taken by the upper code reading camera, which are the posture of the AGV, the relative deviation between the AGV and the ground code, etc.

[0136] It can be seen that the types of calculation results are different according to the types of image devices. The video display method provided in the embodiments of the present application is described below according to the types of image devices:

[0137] In a possible implementation, the image acquisition device of the AGV is an upper barcode reading camera. The upper barcode reading camera is used to read the cargo code. If the first control in the aforementioned step S20 is the first control corresponding to the upper barcode reading camera, the calculation result displayed on the display screen is the posture of the cargo code relative to the AGV and the reading result of the cargo code captured by the upper barcode reading camera.

[0138] The reading result of the cargo code is the code value of the cargo code. The position of the cargo code relative to the AGV can be expressed by the deviation information of the cargo code relative to the AGV, the turntable angle information, the turntable height information, etc.

[0139] For example, Figure 6The figure shows an example of the display of the recognition result of the upper code reading camera provided in the embodiment of the present application. Since the upper code reading camera is selected, it can be seen that the first control in the above step S20 is the upper code reading control. In the video preview area 10, a video containing a two-dimensional code image used to represent the cargo code captured by the upper code reading camera is displayed. Based on the video, the AGV calculates the code value of the cargo code, the angular position deviation of the cargo code relative to the AGV, the turntable angle (the angle at which the turntable of the AGV rotates relative to the AGV body) and the turntable height (the lifting height of the turntable of the AGV relative to the body). The angle position deviation is displayed on the display screen as 3°, the turntable angle is 90°, the turntable height is 300mm, and the code reading result of the cargo code is 10090, wherein the code reading result is the code value obtained by recognizing the two-dimensional code in the video captured by the upper code reading camera.

[0140] By adopting the method of the embodiment of the present application, the user can not only view the video taken by the upper code reading camera through the AGV's display screen, but also view the position of the cargo code relative to the AGV and the code reading result of the cargo code captured by the upper code reading camera. The user can analyze the cause of the abnormality without checking the host computer, thereby improving the efficiency of troubleshooting at the AGV operation site.

[0141] In a possible implementation, the image acquisition device of the AGV is a lower barcode reading camera. The lower barcode reading camera is used for navigation, and can be navigated by a two-dimensional code navigation method or a texture navigation method.

[0142] When navigating by QR code navigation, if the first control in the aforementioned step S20 is the first control corresponding to the lower code reading camera, the calculation results displayed on the display screen are the position of the AGV relative to the ground code captured by the lower code reading camera, the reading result of the ground code and the position of the AGV.

[0143] For example, Figure 7 The first example diagram of the recognition result display of the lower code reading camera provided in the embodiment of the present application is shown. Since the lower code reading control is selected, it can be seen that the first control in the above step S20 is the lower code reading control. The video preview area 10 displays a video of a two-dimensional code image taken by the lower code reading camera and used to represent the ground code. The AGV calculates the code value of the ground code, the posture of the AGV (i.e., the posture of the vehicle), and the deviation of the AGV relative to the ground code (i.e., the relative deviation) based on the video. The posture of the vehicle is displayed on the display screen: (12555, 21005), the relative deviation is: (5, 4, 0), and the reading result of the ground code is 0122502100.

[0144] By adopting the method of the embodiment of the present application, when an AGV using QR code navigation encounters an abnormality or failure, the user can not only view the video captured by the lower code reading camera through the AGV's display screen, but also view the position of the ground code captured by the lower code reading camera relative to the AGV, the reading result of the ground code, and the position of the AGV in its own operating map. The user can analyze the cause of the abnormality without checking the host computer, thereby improving the efficiency of troubleshooting at the AGV operation site.

[0145] In the case of navigating by texture navigation, if the first control in the aforementioned step S20 is the first control corresponding to the lower code reading camera, the calculation result displayed on the display screen is the position of the AGV in its own running map.

[0146] For example, Figure 8 The second example diagram of the recognition result display of the lower code reading camera provided in the embodiment of the present application is shown. Since the lower code reading control is selected, it can be seen that the first control in the above step S20 is the lower code reading control. The video containing real-time texture information captured by the lower code reading camera is displayed in the video preview area 10. The AGV calculates the AGV posture (i.e., the vehicle posture) based on the video, and displays the vehicle posture on the display screen: (12555, 21005, 0).

[0147] By adopting the method of the embodiment of the present application, when an AGV using texture navigation encounters an abnormality or failure, the user can not only view the video taken by the code reading camera through the AGV's display screen, but also view the posture of the AGV. Thus, it is possible to determine whether there is an abnormality in the posture of the AGV through the actual posture of the AGV on site and the posture of the AGV on the display screen without connecting to a host computer, thereby improving the efficiency of troubleshooting at the AGV operation site.

[0148] In a possible implementation, the image acquisition device of the AGV is a depth camera. The depth camera is used for obstacle recognition, obstacle avoidance, and positioning. The following description takes the depth camera as a binocular camera as an example. It can be understood that the left and right eyes of the binocular camera can respectively capture images, usually RGB images, and the three-dimensional coordinate information can be calculated based on the position deviation between corresponding points in the RGB images respectively captured by the left and right eyes of the binocular camera, and the three-dimensional coordinate information of each point can be represented by a depth map.

[0149] In order to facilitate the user to choose to view the RGB image or depth image captured by the binocular camera, if the first control in the aforementioned step S20 is the first control corresponding to the binocular camera, the second control is displayed in the user layer. When the user clicks the second control, the RGB image captured by the binocular camera or the depth image calculated based on the video captured by the binocular camera is displayed.

[0150] Based on the RGB images collected by the left and right eyes of the binocular camera, a stereo image, i.e., a depth image, can be calculated. Based on the depth image, a corresponding obstacle image can be obtained. The obstacle image is marked with obstacles and information about obstacles. Based on this, in a possible implementation, the second control includes a left eye control, a right eye control, a color image control, an obstacle map control, and a depth map control. For example, Figure 9a The figure shows a first example diagram of the depth camera recognition result display provided in an embodiment of the present application. After the user selects the depth camera through the control 20, there are second controls for different types of images in the display interface, such as an obstacle map control 21, a depth map control 22, a color map control 23, a left eye control 24, and a right eye control 25. The user can display the images corresponding to different controls in the video preview area 10 of the display screen by clicking different controls, and the recognition results of the depth camera can also be displayed on the display screen. For example, if there is no obstacle, "Obstacle status: binocular normal" will be displayed.

[0151] It can be understood that when the first control is a depth control, the image captured by the depth camera includes the image captured by the left eye and the image captured by the right eye of the binocular camera, and in the above step S30, the images captured by the left eye and the right eye are scaled at the same time.

[0152] When the user wants to view the video captured by the left eye of the binocular camera, when the user selects the left eye control 24, the AGV responds to the user's selection of the left eye control 24 and displays the image captured by the left eye of the binocular camera of a preset size at the first position of the video layer. Figure 9b The second example diagram of the depth camera recognition result display provided by the embodiment of the present application is shown. Since the left eye control is selected, it can be seen that the first control in the above step S20 is the depth control, and the second control in the above text is the left eye control. The video captured by the left eye of the binocular camera is displayed in the video preview area 10.

[0153] When the user wants to view the video captured by the right eye of the binocular camera, the user selects the right eye control, and the AGV responds to the user's selection of the right eye control by displaying the video captured by the right eye of the binocular camera of a preset size at the first position of the video layer. Fig.9c This is a third example diagram showing the depth camera recognition results provided in an embodiment of the present application. Since the right eye control is selected, it can be seen that the first control in the above step S20 is the depth control, and the second control in the above text is the right eye control. The video captured by the right eye of the binocular camera is displayed in the video preview area 10.

[0154] When the user wants to view the stereoscopic image captured by the binocular camera, the user selects the color picture control. In response to the user's selection of the color picture control, the AGV displays the stereoscopic video calculated based on the RGB images respectively collected by the left and right eyes of the binocular camera, that is, the color picture video, at the first position of the video layer. For example, Figure 9d This is a fourth example diagram showing the depth camera recognition results provided in an embodiment of the present application. The color image control is selected. It can be seen that the first control in the above step S20 is the depth control, and the second control in the above text is the color image control. The color image video calculated based on the RGB images collected by the left and right eyes of the binocular camera is displayed in the video preview area 10.

[0155] By adopting the method of the embodiment of the present application, the user can not only view the stereoscopic image captured by the binocular camera through the AGV display screen, but also view the videos captured by the left and right eyes of the left camera respectively, thereby improving the user experience.

[0156] In a possible implementation, after obtaining the two-dimensional image and the three-dimensional image captured by the binocular camera, an algorithm is used to perform calculations to obtain a depth map and position information of each object in the depth map, thereby determining whether there is an obstacle.

[0157] The depth map will display the distance information of all detected surrounding objects relative to the AGV. The distance information is usually expressed in grayscale values. For example, pixels with higher grayscale values ​​represent objects farther from the AGV, and pixels with lower grayscale values ​​represent objects closer to the AGV. However, since the visual effect of the grayscale image is not obvious when the pixel value changes slightly, it is difficult to intuitively display the distance information of each object. Therefore, in order to facilitate the user to intuitively know the distance information of each object, in a possible implementation, obstacles at different distances in the depth map can be represented by different colors, and obstacles with a distance between the preset first distance and the preset second distance from the AGV are represented by the first color, obstacles with a distance between the preset second distance and the preset third distance from the AGV are represented by the second color, and obstacles with a distance between the preset third distance and the preset fourth distance from the AGV are represented by the third color. It can be understood that each preset distance and each color are set by the user according to needs, and the embodiments of the present application are not limited to this.

[0158] For example, suppose the user presets the depth map so that obstacles with a distance of 1 to 2 meters from the AGV are represented in red, obstacles with a distance of 2 to 3 meters from the AGV are represented in yellow, obstacles with a distance of 3 to 4 meters from the AGV are represented in blue, and objects with a distance of 4 to 5 meters from the AGV are represented in green. The user can view the depth map to understand the distance information of each object and then determine whether there is an obstacle.

[0159] In a possible implementation, the distance between each obstacle and the AGV in the depth map may also be marked in numerical form.

[0160] Although the depth image shows the distance information of all detected obstacles, some obstacles will not affect the driving of the AGV. For example, the depth image may include the distance information of the ceiling, but the ceiling will not affect the driving of the AGV.

[0161] It can be seen that when viewing the depth map, the user cannot intuitively know which obstacles may affect the driving of the AGV. Therefore, in order to facilitate the user to intuitively know the obstacle information, you can also view the obstacle map to understand whether there are obstacles. The obstacle map only marks the obstacles detected and located within the preset obstacle range of the AGV. The preset obstacle range is set by the user based on actual experience, and this application does not limit it.

[0162] Therefore, the second control mentioned above includes an obstacle map control and a depth map control. The user can click the obstacle map control to view the obstacle map obtained based on the video captured by the binocular camera, and the obstacle map is marked with the detected obstacles within the preset obstacle range of the AGV; click the depth image control to view the depth map obtained based on the video captured by the binocular camera, and the depth image is marked with the depth map calculated based on the video captured by the binocular camera, and all detected obstacles are marked in the depth map.

[0163] When the user wants to view the obstacle map, the obstacle map control is selected. In response to the user's selection of the obstacle map control, the AGV displays the obstacle map obtained based on the video captured by the binocular camera at the first position of the video layer. Fig.9e The fifth example diagram of the depth camera recognition result display provided by the embodiment of the present application is shown. The obstacle map control is selected. It can be seen that the first control in the above step S20 is the depth control, and the second control in the above text is the obstacle map control. The obstacle map obtained based on the video captured by the binocular camera is displayed in the video preview area 10. The obstacles detected and located within the preset obstacle range of the AGV are marked in the obstacle map. Obstacles at different distances in the obstacle map can also be marked with different colors. It can be understood that only obstacles located within the preset obstacle range of the AGV are marked in the obstacle map.

[0164] In addition, in order to facilitate the user to know the obstacle closest to the AGV, in a possible implementation manner, the obstacle closest to the AGV may be marked with a dotted frame in the obstacle map.

[0165] When the user wants to view the depth image, he selects the depth map control. In response to the user's selection of the depth map control, the AGV displays the depth map calculated based on the video captured by the binocular camera at the first position of the video layer. All detected obstacles are marked in the depth map. Figure 9f The sixth example diagram of the depth camera recognition result display provided in an embodiment of the present application is shown. The depth map control is selected. It can be seen that the first control in the above step S20 is the depth control, and the second control in the above text is the depth map control. The depth map obtained based on the video captured by the binocular camera is displayed in the video preview area 10.

[0166] In order to improve the user experience, in a possible implementation, when the user selects the second control, annotation information indicating whether the AGV encounters an obstacle will also be displayed on the display screen.

[0167] For example, Figure 9f As shown, the user selects the depth map control, and the distance of the obstacle marked in the depth map displayed in the video preview area is 1088mm, and the obstacle is marked with a dotted box. At the same time, "Is it an obstacle: Yes" is displayed on the display screen to indicate that the AGV has encountered an obstacle. It can be understood that when the user selects the left eye control, the right eye control, the color map control or the obstacle map control, the display screen will also display the marking information used to indicate whether the AGV has encountered an obstacle.

[0168] By adopting the method of the embodiment of the present application, the depth image is processed to obtain an obstacle map, thereby improving the visual effect of the depth map. This makes it easier for users to intuitively locate obstacles in the video by combining the depth map and the obstacle map, thereby improving the efficiency of analyzing abnormalities or faults.

[0169] It is understandable that the above Figure 6~Figure 9f The image displayed in the video preview area 10 is only an example. In practice, the video displayed in the video preview area 10 above is the real picture captured by each image acquisition device. The video captured by the binocular camera displayed on the display screen of the AGV can be black and white or color, depending on the configuration of the binocular camera and user needs. This embodiment of the application is not limited to this.

[0170] In a possible implementation, the user may change the position and size of the transparent area in the user layer by setting, thereby changing the size and position of the video preview area of ​​the overlay layer.

[0171] In the case where the depth camera is a structured light depth camera, since the structured light camera does not have a left eye and a right eye, the second control includes a color image control, a depth image control, and an obstacle image control. In this case, the interface diagram of the AGV display screen is the same as that in the above text. Figure 5a~Figure 5eThe upper barcode reader camera recognition result display example picture and the lower barcode reader camera recognition result display example picture are the same as those in the above text. Figure 6~Figure 7 Similarly, when the user selects the depth camera control, the page displayed is the same as Figure 9a The only difference is that when the depth camera is a structured light depth camera, the left and right eye controls do not exist on the page displayed after the user selects the depth camera control. The other depth camera recognition result display example pictures are the same as those in the above text. Figure 9d~9f Similarly, it can be understood that when the user selects the color picture control, what is displayed on the display screen is the RGB image captured by the structured light depth camera, that is, the color picture video mentioned above.

[0172] In order to implement the video display method provided in the embodiment of the present application, the basic framework diagram of the display software of the AGV display screen provided in the embodiment of the present application is as follows: Fig.10 As shown, first, the main process of the application side draws the user layer through the graphical user interface. After the media library receives the image information of the image acquisition device, it rotates, scales, compresses and converts the format to obtain the video stream of the target video, and then sends the video stream of the target video to the layer module. The layer module embeds the target video into a new layer to obtain the video layer. After the graphics hardware abstraction layer superimposes the user layer and the video layer, it sends them for display through the operating system driver and multimedia interface.

[0173] The driver of the operating system may be a DMR (Direct Rendering Manager) driver or a FB (Frame buffer) driver, which is not limited in the embodiment of the present application.

[0174] Corresponding to the aforementioned first aspect, a second aspect of the present application provides an AGV, the AGV comprising a processor, a display screen, and at least one image acquisition device;

[0175] The image acquisition device is used to capture and send video to the processor;

[0176] The processor is used for, in response to an operation on any of the first controls, acquiring a video captured by an image acquisition device corresponding to the first control targeted by the operation; scaling the acquired video to the preset size to obtain a target video; and sending the target video and a calculation result calculated based on the acquired video to the display screen;

[0177] The display screen is used to display overlay layers, wherein the overlay layers include a user layer at the top layer and a video layer at the bottom layer, a transparent area of ​​a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle; the target video is displayed at the first position of the video layer; and the calculation result is displayed.

[0178] According to the embodiment of the present application, since the overlay layer includes a user layer at the top layer and a video layer at the bottom layer, and a transparent area of ​​a preset size is set at the preset first position of the user layer, and at least one first control corresponding to at least one image acquisition device is set at the preset first position, the user can browse the target video displayed at the first position of the video layer through the transparent area at the first position on the user layer in the overlay layer, and the display screen will also display the calculation results calculated by the AGV based on the acquired video. Compared with the prior art, in which the host computer is mainly connected to the AGV to obtain the video captured by the image acquisition device of the AGV, and then the cause of the fault is analyzed in combination with the video displayed by the host computer, in the present application, the user can view the videos captured by each image acquisition device through the display screen of the AGV while viewing the calculation results calculated by the AGV based on the video captured by the image acquisition device. There is no need for the participation of the host computer, and the cause of the abnormality or fault can be analyzed directly based on the video and calculation results displayed on the display screen of the AGV, thereby improving the efficiency of troubleshooting at the AGV operation site.

[0179] Corresponding to the first aspect, the third aspect of the present application provides a video display device, which is applied to AGV, such as Fig.11 As shown, the device comprises:

[0180] A layer display module 1101 is used to display an overlay layer on the display screen of the AGV, wherein the overlay layer includes a user layer at the top layer and a video layer at the bottom layer, a transparent area of ​​a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle;

[0181] The video acquisition module 1102 is used for acquiring, in response to an operation on any of the first controls, a video captured by an image acquisition device corresponding to the first control targeted by the operation;

[0182] The video scaling module 1103 is used to scale the acquired video to the preset size to obtain the target video;

[0183] The video display module 1104 is used to display the target video at the first position of the video layer, and to display the calculation result obtained by the AGV based on the acquired video through the display screen.

[0184] According to the embodiment of the present application, since the overlay layer includes a user layer at the top layer and a video layer at the bottom layer, and a transparent area of ​​a preset size is set at the preset first position of the user layer, and at least one first control corresponding to at least one image acquisition device is set at the preset first position, the user can browse the target video displayed at the first position of the video layer through the transparent area at the first position on the user layer in the overlay layer, and the display screen will also display the calculation results calculated by the AGV based on the acquired video. Compared with the prior art, in which the host computer is mainly connected to the AGV to obtain the video captured by the image acquisition device of the AGV, and then the cause of the fault is analyzed in combination with the video displayed by the host computer, in the present application, the user can view the videos captured by each image acquisition device through the display screen of the AGV while viewing the calculation results calculated by the AGV based on the video captured by the image acquisition device. There is no need for the participation of the host computer, and the cause of the abnormality or fault can be analyzed directly based on the video and calculation results displayed on the display screen of the AGV, thereby improving the efficiency of troubleshooting at the AGV operation site.

[0185] In a possible implementation, the image acquisition device of the automatic guided vehicle includes an upper code reading camera;

[0186] The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including:

[0187] If the first control targeted by the operation is the first control corresponding to the upper code reading camera, the position of the cargo code photographed by the upper code reading camera relative to the automatic guided vehicle and the code reading result of the cargo code are displayed on the display screen.

[0188] In a possible implementation, the image acquisition device of the automatic guided vehicle includes a lower code reading camera;

[0189] The video display module is specifically used for:

[0190] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is QR code navigation, the position of the ground code photographed by the lower code reading camera relative to the automatic guided vehicle, the reading result of the ground code, and the position of the automatic guided vehicle in its own operating map are displayed on the display screen.

[0191] In a possible implementation, the image acquisition device of the automatic guided vehicle includes a lower code reading camera;

[0192] The video display module is specifically used for:

[0193] If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is texture navigation, the position of the automatic guided vehicle in its own operating map calculated based on the ground texture photographed by the lower code reading camera is displayed through the display screen.

[0194] In a possible implementation, the image acquisition device of the automatic guided vehicle includes a depth camera;

[0195] The video display module is specifically used for:

[0196] Displaying a second control in the user layer;

[0197] In response to an operation on the second control, the target video is displayed at the first position of the video layer, or a calculation result calculated based on the picture captured by the depth camera is displayed at the first position of the video layer.

[0198] In a possible implementation, the depth camera is a binocular camera, and the second control includes a left-eye control, a right-eye control, and a color picture control;

[0199] In response to the operation on the second control, displaying the target video at the first position of the video layer includes:

[0200] In response to an operation on the left-eye control, displaying the video captured by the left eye of the binocular camera of the preset size at the first position of the video layer;

[0201] In response to an operation on the right-eye control, displaying the video captured by the right eye of the binocular camera of the preset size at the first position of the video layer;

[0202] In response to an operation on the color image control, the color image video captured by the binocular camera of the preset size is displayed at the first position of the video layer.

[0203] In a possible implementation manner, the second control includes an obstacle map control and a depth map control;

[0204] In response to the operation on the second control, displaying the target video at the first position of the video layer includes:

[0205] In response to an operation on the obstacle map control, displaying an obstacle map calculated based on the video captured by the depth camera at the first position of the video layer, wherein the obstacle map is marked with detected obstacles within a preset obstacle range of the automatic guided vehicle;

[0206] In response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and all detected obstacles are marked in the depth map.

[0207] In a possible implementation, the device further includes:

[0208] An information display module is used to display, in response to an operation on the second control, labeling information indicating whether the automatic guided vehicle encounters an obstacle through the display screen.

[0209] In a fourth aspect of the present application, an electronic device is also provided, such as Fig.12 As shown, including:

[0210] Memory 1201, used for storing computer programs;

[0211] The processor 1202 is used to implement any of the above-mentioned video display methods when executing the program stored in the memory.

[0212] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0213] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0214] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned video display methods are implemented.

[0215] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the video display methods in the above embodiments.

[0216] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0217] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0218] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the automatic guided vehicle and device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0219] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A video display method, characterized in that: Applied to an automatic guided vehicle, the method comprises: Displaying an overlay layer on a display screen of the automatic guided vehicle, wherein the overlay layer includes a user layer at a top layer and a video layer at a bottom layer, a transparent area of ​​a preset size is provided at a preset first position of the user layer, at least one first control is provided at a preset second position of the user layer, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle; In response to an operation on any of the first controls, obtaining a video captured by an image acquisition device corresponding to the first control targeted by the operation; Scaling the acquired video to the preset size to obtain the target video; The target video is displayed at the first position of the video layer, and a calculation result obtained by the automatic guided vehicle based on the acquired video is displayed on the display screen.

2. The method according to claim 1, characterized in that The image acquisition device of the automatic guided vehicle includes an upper code reading camera; The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including: If the first control targeted by the operation is the first control corresponding to the upper code reading camera, the position of the cargo code photographed by the upper code reading camera relative to the automatic guided vehicle and the code reading result of the cargo code are displayed on the display screen.

3. The method according to claim 1, characterized in that The image acquisition device of the automatic guided vehicle includes a lower code reading camera; The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including: If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is QR code navigation, the position of the ground code photographed by the lower code reading camera relative to the automatic guided vehicle, the reading result of the ground code, and the position of the automatic guided vehicle in its own operating map are displayed on the display screen.

4. The method according to claim 1, characterized in that: The image acquisition device of the automatic guided vehicle includes a lower code reading camera; The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including: If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is texture navigation, the position of the automatic guided vehicle in its own operating map calculated based on the ground texture photographed by the lower code reading camera is displayed through the display screen.

5. The method according to claim 1, characterized in that The image acquisition device of the automatic guided vehicle includes a depth camera; The step of displaying the target video at the first position of the video layer, and displaying the calculation result obtained by the automatic guided vehicle based on the acquired video through the display screen, includes: Displaying a second control in the user layer; In response to an operation on the second control, the target video is displayed at the first position of the video layer, or a calculation result calculated based on the picture captured by the depth camera is displayed at the first position of the video layer.

6. The method according to claim 5, characterized in that The depth camera is a binocular camera, and the second control includes a left-eye control, a right-eye control, and a color picture control; In response to the operation on the second control, displaying the target video at the first position of the video layer includes: In response to an operation on the left-eye control, displaying the video captured by the left eye of the binocular camera of the preset size at the first position of the video layer; In response to an operation on the right-eye control, displaying the video captured by the right eye of the binocular camera of the preset size at the first position of the video layer; In response to an operation on the color image control, the color image video captured by the binocular camera of the preset size is displayed at the first position of the video layer.

7. The method according to claim 5, characterized in that The second control includes an obstacle map control and a depth map control; In response to the operation on the second control, displaying the target video at the first position of the video layer includes: In response to an operation on the obstacle map control, displaying an obstacle map calculated based on the video captured by the depth camera at the first position of the video layer, wherein the obstacle map is marked with detected obstacles within a preset obstacle range of the automatic guided vehicle; In response to an operation on the depth map control, a depth map calculated based on the video captured by the depth camera is displayed at the first position of the video layer, and all detected obstacles are marked in the depth map.

8. The method according to claim 6 or 7, characterized in that: The method further comprises: In response to the operation on the second control, labeling information indicating whether the automatic guided vehicle encounters an obstacle is displayed on the display screen.

9. An automatic guided vehicle, characterized in that: The automatic guided vehicle includes a processor, a display screen and at least one image acquisition device; The image acquisition device is used to capture and send video to the processor; The processor is configured to, in response to an operation on any first control, obtain a video captured by an image acquisition device corresponding to the first control targeted by the operation; and scale the obtained video to a preset size to obtain a target video; Sending the target video and a calculation result calculated based on the acquired video to the display screen; The display screen is used to display overlay layers, wherein the overlay layers include a user layer at the top layer and a video layer at the bottom layer, a transparent area of ​​a preset size is set at a preset first position of the user layer, and at least one first control is set at a preset second position of the user layer, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle; the target video is displayed at the first position of the video layer; and the calculation result is displayed.

10. A video display device, characterized in that: Applied to an automatic guided vehicle, the device comprises: A layer display module, configured to display an overlay layer on a display screen of the automatic guided vehicle, wherein the overlay layer includes a user layer at a top layer and a video layer at a bottom layer, a transparent area of ​​a preset size is provided at a preset first position of the user layer, and at least one first control is provided at a preset second position of the user layer, and the first control corresponds to at least one image acquisition device of the automatic guided vehicle; A video acquisition module, configured to, in response to an operation on any of the first controls, acquire a video captured by an image acquisition device corresponding to the first control targeted by the operation; A video scaling module is used to scale the acquired video to the preset size to obtain a target video; A video display module is used to display the target video at the first position of the video layer, and to display the calculation results obtained by the automatic guided vehicle based on the acquired video through the display screen.

11. The device according to claim 10, characterized in that The image acquisition device of the automatic guided vehicle includes an upper code reading camera; The display screen displays the calculation result obtained by the automatic guided vehicle based on the acquired video, including: If the first control targeted by the operation is the first control corresponding to the upper code reading camera, the position of the cargo code photographed by the upper code reading camera relative to the automatic guided vehicle and the code reading result of the cargo code are displayed on the display screen; and / or The image acquisition device of the automatic guided vehicle includes a lower code reading camera; The video display module is specifically used for: If the first control targeted by the operation is the first control corresponding to the lower code reading camera, and the navigation mode of the automatic guided vehicle is QR code navigation, the position of the ground code photographed by the lower code reading camera relative to the automatic guided vehicle, the reading result of the ground code, and the position of the automatic guided vehicle in its own running map are displayed on the display screen; and / or The image acquisition device of the automatic guided vehicle includes a lower code reading camera; The video display module is specifically used for: If the first control targeted by the operation is the first control corresponding to the lower barcode reading camera, and the navigation mode of the automatic guided vehicle is texture navigation, the position of the automatic guided vehicle in its own operation map calculated based on the ground texture photographed by the lower barcode reading camera is displayed on the display screen; and / or The image acquisition device of the automatic guided vehicle includes a depth camera; The video display module is specifically used for: Displaying a second control in the user layer; In response to an operation on the second control, displaying the target video at the first position of the video layer, or displaying a calculation result calculated based on the picture captured by the depth camera at the first position of the video layer; and / or The depth camera is a binocular camera, and the second control includes a left-eye control, a right-eye control, and a color picture control; In response to the operation on the second control, displaying the target video at the first position of the video layer includes: In response to an operation on the left-eye control, displaying the video captured by the left eye of the binocular camera of the preset size at the first position of the video layer; In response to an operation on the right-eye control, displaying the video captured by the right eye of the binocular camera of the preset size at the first position of the video layer; In response to an operation on the color image control, displaying the color image video captured by the binocular camera of the preset size at the first position of the video layer; and / or The second control includes an obstacle map control and a depth map control; In response to the operation on the second control, displaying the target video at the first position of the video layer includes: In response to an operation on the obstacle map control, displaying an obstacle map calculated based on the video captured by the depth camera at the first position of the video layer, wherein the obstacle map is marked with detected obstacles within a preset obstacle range of the automatic guided vehicle; In response to an operation on the depth map control, displaying a depth map calculated based on the video captured by the depth camera at the first position of the video layer, wherein all detected obstacles are marked in the depth map; and / or The device also includes: An information display module is used to display, through the display screen, labeling information indicating whether the automatic guided vehicle encounters an obstacle in response to an operation on the second control.

12. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images