Display screen information collection method, device and medium

Abstract

The present application discloses a display screen information acquisition method, device and medium. The display screen information acquisition method is applied to a display screen information acquisition device. The display screen information acquisition device includes an image acquisition device, a wireless module and a processing module. The method includes: obtaining a captured image of a display screen to be acquired through the image acquisition device, and dividing the captured image into a first partial image, a second partial image, a third partial image and a fourth partial image; flipping the second partial image left to right, flipping the third partial image upside down, and flipping the fourth partial image upside down and left to right; determining a preset convolutional neural network model, and determining the screen corner coordinates of the display screen to be acquired corresponding to the first partial image, the flipped second partial image, the flipped third partial image and the flipped fourth partial image respectively through the preset convolutional neural network model; and restoring the captured image to a display image according to perspective transformation and the screen corner coordinates.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a display screen information acquisition method, device and medium. Background Art

[0002] In some areas, such as the railway CTC dispatching centralized control system, it is not allowed to connect to external systems. If you want to obtain the display screen information of such a system, you can only set up an image acquisition device from the outside, and then use the image acquisition device to obtain the captured image of the display screen to be captured, and then perform distortion correction, reflection processing and other steps on the captured image to restore the display image of the display screen to be captured. In order not to affect the staff's use of the display screen, the image acquisition device must be placed close to the display screen to be captured, or even close to the display screen. However, once the camera of the existing image acquisition device cannot face the display screen to be captured, it is easy to cause image distortion and other phenomena.

[0003] Therefore, there is an urgent need for a display screen information acquisition method that can obtain a normal image even if the camera of the image acquisition device is not facing the display screen. Summary of the invention

[0004] In order to solve the above problems, the present application proposes a display screen information acquisition method, device and medium, wherein the display screen information acquisition method is applied to a display screen information acquisition device, wherein the display screen information acquisition device includes an image acquisition device, a wireless module and a processing module; the method includes:

[0005] The captured image of the display screen to be captured is acquired through the image acquisition device, and the captured image is divided into a first partial image, a second partial image, a third partial image, and a fourth partial image, so that the first partial image is located at the upper left of the captured image, the second partial image is located at the upper right of the captured image, the third partial image is located at the lower left of the captured image, and the fourth partial image is located at the lower right of the captured image; the second partial image is flipped left to right, the third partial image is flipped upside down, and the fourth partial image is flipped upside down and left to right; a preset convolutional neural network model is determined, and the screen corner coordinates of the display screen to be captured corresponding to the first partial image, the flipped second partial image, the flipped third partial image, and the flipped fourth partial image are respectively determined through the preset convolutional neural network model; according to the perspective transformation and the screen corner coordinates, the captured image is restored to a display image.

[0006] In one example, the screen angular coordinates of the display screen to be captured corresponding to the first local image, the flipped second local image, the flipped third local image and the flipped fourth local image are respectively determined by the preset convolutional neural network model, specifically including: preprocessing the first local image, the flipped second local image, the flipped third local image and the flipped fourth local image, and the preprocessing method includes: at least one of defocus repair and image unevenness elimination; converting the preprocessed image into a three-channel image to be input with a pixel number of 256*256.

[0007] In one example, after converting the preprocessed image into a three-channel input image with 256*256 pixels, the method further includes: determining the horizontal coordinate and the vertical coordinate in the screen corner coordinate through the convolution layer and the fully connected layer in the preset convolutional neural network model.

[0008] In one example, the horizontal coordinates in the screen corner coordinates are determined by the convolution layer and the fully connected layer in the preset convolutional neural network model, specifically including: inputting the image to be input into the preset convolutional neural network model, downsampling the image information of the image to be input through the first horizontal coordinate convolution layer, and converting it into a first horizontal coordinate image matrix with 32 channels and 128*64 pixels; downsampling the first horizontal coordinate image matrix through the second horizontal coordinate convolution layer, and converting it into a second horizontal coordinate image matrix with 256 channels and 64*16 pixels; downsampling the second horizontal coordinate image matrix through the third horizontal coordinate convolution layer, and converting it into a third horizontal coordinate image matrix with 256 channels and 32*4 pixels; downsampling the second horizontal coordinate image matrix through the fourth horizontal coordinate convolution layer, and converting it into a third horizontal coordinate image matrix with 256 channels and 32*4 pixels; downsampling the second horizontal coordinate image matrix through the fourth horizontal coordinate convolution layer, and converting it into a third horizontal coordinate image matrix with 256 channels and 32*4 pixels. The convolution layer downsamples the third horizontal coordinate image matrix and converts it into a fourth horizontal coordinate image matrix with 256 channels and 16*1 pixels; the first horizontal coordinate convolution layer, the second horizontal coordinate convolution layer, the third horizontal coordinate convolution layer, and the fourth horizontal coordinate convolution layer use 3*5-dimensional rectangular convolution kernels, and perform maximum pooling through 2*4 pooling convolution kernels; the fourth horizontal coordinate image information is converted into a fifth horizontal coordinate image matrix with 16 channels and 256*1 dimensions through the first horizontal coordinate fully connected layer; the fifth horizontal coordinate image matrix is ​​converted into a horizontal coordinate vector group with 256 channels and 16*1 dimensions through the second horizontal coordinate fully connected layer; the horizontal coordinate in the screen corner coordinate of the display screen to be collected is determined through the normalized exponential function and the horizontal coordinate vector group.

[0009] In one example, the vertical coordinate in the screen corner coordinate is determined by the convolution layer and the fully connected layer in the preset convolutional neural network model, specifically including: inputting the image to be input into the preset convolutional neural network model, downsampling the image information of the image to be input through the first vertical coordinate convolution layer, and converting it into a first vertical coordinate image matrix with 32 channels and 64*128 pixels; downsampling the first vertical coordinate image matrix through the second horizontal coordinate convolution layer, and converting it into a second vertical coordinate image matrix with 256 channels and 16*64 pixels; downsampling the second vertical coordinate image matrix through the third horizontal coordinate convolution layer, and converting it into a third vertical coordinate image matrix with 256 channels and 4*32 pixels; downsampling the second vertical coordinate image matrix through the fourth horizontal coordinate convolution layer, and converting it into a third vertical coordinate image matrix with 256 channels and 4*32 pixels; downsampling the second vertical coordinate image matrix through the fourth horizontal coordinate convolution layer, and converting it into a third vertical coordinate image matrix with 256 channels and 4*32 pixels. The third ordinate image matrix is ​​downsampled by a standard convolution layer and converted into a fourth ordinate image matrix with 256 channels and 1*16 pixels; the first ordinate convolution layer, the second ordinate convolution layer, the third ordinate convolution layer, and the fourth ordinate convolution layer adopt a 5*3-dimensional rectangular convolution kernel, and perform maximum pooling through a 4*2 pooling convolution kernel; the fourth ordinate image information is converted into a fifth ordinate image matrix with 16 channels and 256*1 dimensions through a first ordinate fully connected layer; the fifth ordinate image matrix is ​​converted into a 256-channel, 16*1-dimensional ordinate vector group through a second ordinate fully connected layer; the ordinate in the screen corner coordinate is determined by a normalized exponential function and the ordinate vector group.

[0010] In one example, the method of using perspective transformation to restore the captured image to a display image through the screen corner coordinates specifically includes: converting the screen corner coordinates into global image corner coordinates based on the flipping relationship of the local image; and restoring the captured image to the display image of the display screen to be captured based on the perspective transformation method and the corner coordinates of the global image.

[0011] In one example, the method further includes: determining that the partial image does not include the screen corner coordinates of the display screen to be captured; and sending an alarm message through the wireless module so that the image capture device can be moved according to the alarm message.

[0012] In one example, the method further includes: determining that the computing workload exceeds a preset threshold, transmitting the captured image to a server for computing via the wireless transmission module; and receiving the displayed image restored by the server via the wireless transmission module.

[0013] The present application also provides a display screen information collection device, including:

[0014] At least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute: acquiring a captured image of the display screen to be captured through an image acquisition device, and dividing the captured image into a first partial image, a second partial image, a third partial image, and a fourth partial image, so that the first partial image is located at the upper left of the captured image, the second partial image is located at the upper right of the captured image, the third partial image is located at the lower left of the captured image, and the fourth partial image is located at the lower right of the captured image; flipping the second partial image left to right, flipping the third partial image upside down, and flipping the fourth partial image upside down and left to right; determining a preset convolutional neural network model, and determining the screen corner coordinates of the display screen to be captured corresponding to the first partial image, the flipped second partial image, the flipped third partial image, and the flipped fourth partial image respectively through the preset convolutional neural network model; restoring the captured image to a display image according to the perspective transformation and the screen corner coordinates.

[0015] The present application also provides a non-volatile computer storage medium, storing computer executable instructions, wherein the computer executable instructions are configured to: obtain a captured image of a display screen to be captured through an image acquisition device, and divide the captured image into a first partial image, a second partial image, a third partial image, and a fourth partial image, so that the first partial image is located at the upper left of the captured image, the second partial image is located at the upper right of the captured image, the third partial image is located at the lower left of the captured image, and the fourth partial image is located at the lower right of the captured image; flip the second partial image left to right, flip the third partial image upside down, and flip the fourth partial image upside down and left to right; determine a preset convolutional neural network model, and determine the screen corner coordinates of the display screen to be captured corresponding to the first partial image, the flipped second partial image, the flipped third partial image, and the flipped fourth partial image through the preset convolutional neural network model; restore the captured image to a display image according to the perspective transformation and the screen corner coordinates.

[0016] The method proposed in the present application enables normal images to be acquired even if the camera of the image acquisition device is not facing the display screen, thereby solving the problem of being unable to acquire the contents of the display screen in some systems with high security alerts. As a result, even if the system with high security alerts is not connected to other systems, the external system can acquire the display image of the display screen to be acquired by setting up an image acquisition device. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 This is a schematic diagram of a display screen information collection method in an embodiment of the present application;

[0019] Figure 2 This is a schematic diagram of a process of determining screen corner coordinates through a convolutional neural network model in an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of a display screen information collection device in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] The implementation of the analysis method involved in the embodiment of the present application can be a terminal device or a server, and the present application does not impose any special restrictions on this. For the convenience of understanding and description, the following embodiments are described in detail using the terminal device as an example. The following is a detailed description of the technical solutions provided by each embodiment of the present application in conjunction with the accompanying drawings.

[0023] like Figure 1 As shown, the embodiment of the present application provides a display screen information acquisition method, which is applied to a display screen information acquisition device, and the display screen information acquisition device includes an image acquisition device, a wireless module, and a processing module. Among them, the image acquisition device is used to collect image data of the display screen, and can use devices such as cameras and high-definition cameras. The wireless module is used for data interaction with the cloud, and the processing module is used for local processing of image data. The display screen information acquisition method includes:

[0024] S101: Acquire a captured image of the display screen to be captured by the image acquisition device, and divide the captured image into a first partial image, a second partial image, a third partial image, and a fourth partial image, so that the first partial image is located at the upper left of the captured image, the second partial image is located at the upper right of the captured image, the third partial image is located at the lower left of the captured image, and the fourth partial image is located at the lower right of the captured image.

[0025] First, the display screen to be captured needs to be captured by an image capture device to obtain a captured image of the display screen to be captured, and then the captured image is divided into four equal parts: a first partial image, a second partial image, a third partial image, and a fourth partial image. The first partial image is located at the upper left corner of the original captured image, the second partial image is located at the upper right corner of the original captured image, the third partial image is located at the lower left corner of the original captured image, and the fourth partial image is located at the lower right corner of the original captured image.

[0026] S102: Flip the second partial image left to right, flip the third partial image upside down, and flip the fourth partial image upside down and left to right.

[0027] After the captured image is divided into four partial images by a cross, since the display screen to be captured is a rectangle, the display screen in the captured image will present a trapezoidal shape. The four corners of the display image of the default display screen are in the captured image, and each partial image has a corner of the display image of the display screen to be captured. On this basis, the second partial image is flipped left and right, the third partial image is flipped up and down, and the fourth partial image is flipped up and down and left and right. In this way, each of the four partial images will have a corner of the trapezoid in the global image, and the orientation is basically the same.

[0028] S103: Determine a preset convolutional neural network model, and use the preset convolutional neural network model to respectively determine the screen corner coordinates of the display screen to be captured corresponding to the first local image, the flipped second local image, the flipped third local image, and the flipped fourth local image.

[0029] First, it is necessary to determine the pre-set and trained convolutional neural network model. The input of the convolutional neural network model here is the local image, and the output is the screen corner coordinates of the display screen to be collected in the local image, that is, the vertex coordinates of the trapezoid corner in the local image. Here we use the upper left corner of the first local image and the flipped second, third, and fourth local images as the origin to establish a coordinate system. Input the first local image and the flipped second, third, and fourth local images into the convolutional neural network model, and then output the vertex coordinates of the trapezoid corner in the flipped local image. For convenience, the vertex coordinates in the first, flipped second, third, and fourth local images can be called (u1, v1), (u2, v2), (u3, v3), and (u4, v4), respectively.

[0030] S104: Restoring the captured image to a display image according to the perspective transformation and the screen corner coordinates.

[0031] After obtaining the display content of the display screen to be captured contained in the flipped partial image, the captured image is restored to the display image using a perspective transformation method, thus obtaining the display content of the display screen through the external image capture device.

[0032] In one embodiment, after the local image is input into the convolutional neural network model, the local image needs to be preprocessed first. According to the local image, it is determined whether the image contains lens defocus, image unevenness, etc., and then the local image is processed by a corresponding method to reduce the impact of vertex recognition in the subsequent process. The preprocessed image is converted into a three-channel (RGB) image with a pixel number of 256*256 to facilitate the convolutional neural network model to process the input image in the subsequent process and reduce the amount of calculation.

[0033] Furthermore, after the local image is preprocessed and converted into the image to be input, the convolutional neural network model processes the input image through a convolutional layer and a fully connected layer to determine the horizontal and vertical coordinates of the vertices in the local image.

[0034] like Figure 2As shown, the convolutional layer downsampling algorithm of the neural network uses two modes, 4x4->2x1 and 4x4->1x2, for the two network channels X and Y respectively. This recursion continues until the 256x256 square matrix is ​​finally compressed into two 16x1 and 1x16 dimensional vector groups of 256 channels on the left and right. Specifically, when determining the horizontal coordinates of the screen corner coordinates in the local image, the image to be input needs to be input into the convolutional neural network model, and the image information of the image to be input is downsampled through the first horizontal coordinate convolution layer, i.e., the Xconv1 convolution layer in the figure, and converted into a first horizontal coordinate image matrix with 32 channels and 128*64 pixels; the first horizontal coordinate image matrix is ​​downsampled through the second horizontal coordinate convolution layer, i.e., the Xconv2 convolution layer in the figure, and converted into a second horizontal coordinate image matrix with 256 channels and 64*16 pixels; the second horizontal coordinate convolution layer is downsampled through the third horizontal coordinate convolution layer, i.e., the Xconv3 convolution layer in the figure. The first horizontal coordinate convolution layer, the second horizontal coordinate convolution layer, the third horizontal coordinate convolution layer, and the fourth horizontal coordinate convolution layer all use a 3*5-dimensional rectangular convolution kernel, and perform maximum pooling through a 2*4 pooling convolution kernel, so that the number of rows of the image matrix after convolution is changed to half of that before convolution, and the number of columns is changed to one quarter of that before convolution. Then, through the first fully connected layer, that is, the XFN256 convolution layer in the figure, the fourth horizontal coordinate image information is converted into a 256*1 fifth horizontal coordinate image matrix; through the second fully connected layer, that is, the XFN16 convolution layer in the figure, the fifth horizontal coordinate image matrix is ​​converted into a 16*1 sixth horizontal coordinate image matrix; through the normalized exponential function and the sixth horizontal coordinate image matrix, the horizontal coordinates of the vertices in the local image, that is, the values ​​of u1, u2, u3, and u4, are determined.

[0035] Based on the same idea, when determining the ordinates of the vertices in the local image, the image to be input is input into the preset convolutional neural network model, and the image information of the image to be input is downsampled through the first ordinate convolution layer, that is, the Yconv1 convolution layer in the figure, and converted into a first ordinate image matrix with 32 channels and 64*128 pixels; through the second abscissa convolution layer, that is, the Yconv2 convolution layer in the figure, the first ordinate image matrix is ​​downsampled and converted into a second ordinate image matrix with 256 channels and 16*64 pixels; through the third abscissa convolution layer, that is, the Yconv3 convolution layer in the figure, the second ordinate image matrix is ​​downsampled and converted into a second ordinate image matrix with 256 channels and 16*64 pixels. The first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer all use a 5*3-dimensional rectangular convolution kernel, and perform maximum pooling through a 4*2 pooling convolution kernel, so that the number of rows of the image matrix after convolution is changed to one-fourth of that before convolution, and the number of columns is changed to one-half of that before convolution. Then, through the first fully connected layer, that is, the YFN256 convolution layer in the figure, the fourth ordinate image information is converted into a 256*1 fifth ordinate image matrix; through the second fully connected layer, that is, the YFN16 convolution layer in the figure, the fifth ordinate image matrix is ​​converted into a 16*1 sixth ordinate image matrix; through the normalized exponential function and the sixth ordinate image matrix, the image angular ordinate of the display screen to be collected is determined, that is, the values ​​of v1, v2, v3, and v4 are determined. Among them, the convolution kernels of the X and Y channels are 3*5 and 5*3 rectangular convolution kernels respectively.

[0036] In one embodiment, after obtaining the vertex coordinates of the four corners of the trapezoid in the flipped partial image, the vertex coordinates are restored to the vertex coordinates before flipping according to the flipping relationship and the size of the image. That is, (x1, y1), (x2, y2), (x3, y3), (x4, y4) in the original captured image are determined by (u1, v1), (u2, v2), (u3, v3), (u4, v4) and then the captured image is restored to the display image of the display screen to be captured according to the perspective transformation method.

[0037] In one embodiment, because the image acquisition device may be placed in a remote location, the captured image may be severely deformed, resulting in a phenomenon that one of the four partial images obtained by cross-division does not contain a corner of the trapezoid. At this time, an alarm message can be sent through the wireless transmission module so that the image acquisition device can be moved according to the alarm message. The moving method can be moved by staff or by installing a guide rail under the image acquisition device, which is not limited here. If this situation still occurs after the movement, the wireless transmission module will continue to alarm until this situation disappears.

[0038] In one embodiment, due to the excessive amount of calculations that may occur during the image processing and vertex recognition process, if the display screen information acquisition task requires strict real-time performance, the machine may respond slowly. To avoid this, the processing module can monitor the computing workload in the processing module in real time. If the computing workload exceeds the preset threshold, the captured image is transmitted to the cloud through the wireless module, and the captured image is processed through the cloud, thereby alleviating the computing workload of the processing module. Then, the display image restored from the cloud is received through the wireless transmission module.

[0039] like Figure 3 As shown, the embodiment of the present application also provides a display screen information collection device, including:

[0040] at least one processor; and,

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0043] Acquire a captured image of the display screen to be captured by an image acquisition device, and divide the captured image into a first partial image, a second partial image, a third partial image, and a fourth partial image, so that the first partial image is located at the upper left of the captured image, the second partial image is located at the upper right of the captured image, the third partial image is located at the lower left of the captured image, and the fourth partial image is located at the lower right of the captured image;

[0044] Flipping the second partial image left to right, flipping the third partial image upside down, and flipping the fourth partial image upside down and left to right;

[0045] Determine a preset convolutional neural network model, and determine the screen corner coordinates of the display screen to be collected corresponding to the first partial image, the flipped second partial image, the flipped third partial image, and the flipped fourth partial image respectively through the preset convolutional neural network model;

[0046] The captured image is restored to a display image according to the perspective transformation and the screen corner coordinates.

[0047] The embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0048] Acquire a captured image of the display screen to be captured by an image acquisition device, and divide the captured image into a first partial image, a second partial image, a third partial image, and a fourth partial image, so that the first partial image is located at the upper left of the captured image, the second partial image is located at the upper right of the captured image, the third partial image is located at the lower left of the captured image, and the fourth partial image is located at the lower right of the captured image;

[0049] Flipping the second partial image left to right, flipping the third partial image upside down, and flipping the fourth partial image upside down and left to right;

[0050] Determine a preset convolutional neural network model, and determine the screen corner coordinates of the display screen to be collected corresponding to the first partial image, the flipped second partial image, the flipped third partial image, and the flipped fourth partial image respectively through the preset convolutional neural network model;

[0051] The captured image is restored to a display image according to the perspective transformation and the screen corner coordinates.

[0052] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0053] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0054] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0055] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0056] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0057] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0058] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0059] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0060] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0061] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0062] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A display screen information acquisition method, applied to a display screen information acquisition device, wherein the display screen information acquisition device comprises an image acquisition device, a wireless module, and a processing module. It is characterized in that The method comprises: Acquire a captured image of the display screen to be captured by the image acquisition device, and divide the captured image into a first partial image, a second partial image, a third partial image, and a fourth partial image, so that the first partial image is located at the upper left of the captured image, the second partial image is located at the upper right of the captured image, the third partial image is located at the lower left of the captured image, and the fourth partial image is located at the lower right of the captured image; Flipping the second partial image left to right, flipping the third partial image upside down, and flipping the fourth partial image upside down and left to right; Determine a preset convolutional neural network model, and determine the screen corner coordinates of the display screen to be collected corresponding to the first partial image, the flipped second partial image, the flipped third partial image, and the flipped fourth partial image respectively through the preset convolutional neural network model; The captured image is restored to a display image according to the perspective transformation and the screen corner coordinates.

2. The method according to claim 1, It is characterized in that Determining the screen corner coordinates of the display screen to be collected corresponding to the first partial image, the flipped second partial image, the flipped third partial image, and the flipped fourth partial image respectively by the preset convolutional neural network model specifically includes: Preprocessing the first partial image, the flipped second partial image, the flipped third partial image, and the flipped fourth partial image, wherein the preprocessing method includes at least one of: out-of-focus repair and image unevenness elimination; The preprocessed image is converted into a three-channel image to be input with a pixel number of 256*256.

3. The method according to claim 2, It is characterized in that After converting the preprocessed image into a three-channel image to be input with a pixel number of 256*256, the method further includes: The horizontal coordinate and the vertical coordinate in the screen corner coordinate are determined by the convolution layer and the fully connected layer in the preset convolutional neural network model.

4. The method according to claim 3, It is characterized in that Determining the horizontal coordinate in the screen corner coordinate by using the convolution layer and the fully connected layer in the preset convolutional neural network model specifically includes: Input the image to be input into the preset convolutional neural network model, downsample the image information of the image to be input through the first horizontal coordinate convolution layer, and convert it into a first horizontal coordinate image matrix with 32 channels and 128*64 pixels; Down-sampling the first horizontal coordinate image matrix through a second horizontal coordinate convolution layer to convert it into a second horizontal coordinate image matrix with 256 channels and 64*16 pixels; Down-sampling the second horizontal coordinate image matrix through a third horizontal coordinate convolution layer to convert it into a third horizontal coordinate image matrix with 256 channels and 32*4 pixels; Down-sampling the third horizontal coordinate image matrix through a fourth horizontal coordinate convolution layer to convert it into a fourth horizontal coordinate image matrix with 256 channels and 16*1 pixels; The first horizontal coordinate convolution layer, the second horizontal coordinate convolution layer, the third horizontal coordinate convolution layer, and the fourth horizontal coordinate convolution layer use a 3*5 dimensional rectangular convolution kernel, and perform maximum pooling through a 2*4 pooling convolution kernel; The fourth abscissa image information is converted into a fifth abscissa image matrix with 16 channels and 256*1 dimensions through a first abscissa fully connected layer; The fifth horizontal coordinate image matrix is ​​converted into a 256-channel, 16*1-dimensional horizontal coordinate vector group through a second horizontal coordinate fully connected layer; The abscissa in the screen angle coordinates of the display screen to be collected is determined by using the normalized exponential function and the abscissa vector group.

5. The method according to claim 3, It is characterized in that Determining the vertical coordinate in the screen corner coordinate by using the convolution layer and the fully connected layer in the preset convolutional neural network model specifically includes: Input the image to be input into the preset convolutional neural network model, downsample the image information of the image to be input through the first ordinate convolution layer, and convert it into a first ordinate image matrix with 32 channels and 64*128 pixels; Down-sampling the first ordinate image matrix through a second abscissa convolution layer to convert it into a second ordinate image matrix with 256 channels and 16*64 pixels; Down-sampling the second ordinate image matrix through a third abscissa convolution layer to convert it into a third ordinate image matrix with 256 channels and 4*32 pixels; Down-sampling the third ordinate image matrix through a fourth abscissa convolution layer to convert it into a fourth ordinate image matrix with 256 channels and 1*16 pixels; The first ordinate convolution layer, the second ordinate convolution layer, the third ordinate convolution layer, and the fourth ordinate convolution layer use a 5*3 dimensional rectangular convolution kernel, and perform maximum pooling through a 4*2 pooling convolution kernel; The fourth ordinate image information is converted into a fifth ordinate image matrix with 16 channels and 256*1 dimensions through a first ordinate fully connected layer; The fifth ordinate image matrix is ​​converted into a 256-channel, 16*1-dimensional ordinate vector group through a second ordinate fully connected layer; The ordinate in the screen corner coordinate is determined by using a normalized exponential function and the ordinate vector group.

6. The method according to claim 1, It is characterized in that The method of using perspective transformation to restore the captured image to a display image through the screen corner coordinates specifically includes: Converting the screen angular coordinates into global image angular coordinates according to the flipping relationship of the local image; According to the perspective transformation method and the angular coordinates of the global image, the captured image is restored to the display image of the display screen to be captured.

7. The method according to claim 1, It is characterized in that The method further comprises: Determining that the partial image does not include the screen corner coordinates of the display screen to be captured; The wireless module sends out an alarm message so that the image acquisition device can be moved according to the alarm message.

8. The method according to claim 1, It is characterized in that The method further comprises: Determining that the computing workload exceeds a preset threshold, transmitting the captured image to a server for computing via the wireless transmission module; The display image restored by the server is received through the wireless transmission module.

9. A display screen information collection device, It is characterized in that include: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute: Acquire a captured image of the display screen to be captured by an image acquisition device, and divide the captured image into a first partial image, a second partial image, a third partial image, and a fourth partial image, so that the first partial image is located at the upper left of the captured image, the second partial image is located at the upper right of the captured image, the third partial image is located at the lower left of the captured image, and the fourth partial image is located at the lower right of the captured image; Flipping the second partial image left to right, flipping the third partial image upside down, and flipping the fourth partial image upside down and left to right; Determine a preset convolutional neural network model, and determine the screen corner coordinates of the display screen to be collected corresponding to the first partial image, the flipped second partial image, the flipped third partial image, and the flipped fourth partial image respectively through the preset convolutional neural network model; The captured image is restored to a display image according to the perspective transformation and the screen corner coordinates.

10. A non-volatile computer storage medium storing computer executable instructions, It is characterized in that The computer executable instructions are configured to: Acquire a captured image of the display screen to be captured by an image acquisition device, and divide the captured image into a first partial image, a second partial image, a third partial image, and a fourth partial image, so that the first partial image is located at the upper left of the captured image, the second partial image is located at the upper right of the captured image, the third partial image is located at the lower left of the captured image, and the fourth partial image is located at the lower right of the captured image; Flipping the second partial image left to right, flipping the third partial image upside down, and flipping the fourth partial image upside down and left to right; Determine a preset convolutional neural network model, and determine the screen corner coordinates of the display screen to be collected corresponding to the first partial image, the flipped second partial image, the flipped third partial image, and the flipped fourth partial image respectively through the preset convolutional neural network model; The captured image is restored to a display image according to the perspective transformation and the screen corner coordinates.

Images