A touch screen object positioning method and device based on artificial intelligence

Abstract

The present invention discloses a touch screen object positioning method and device based on artificial intelligence, wherein the method comprises: obtaining an original image that at least completely covers the touch screen, and extracting the touch screen screen from it to obtain a screen image to be identified; inputting the screen image to be identified into a deep learning network model to obtain the position information of the object to be identified on the screen image to be identified; generating a data packet based on the position information of the identified object, and sending the data packet to the touch screen; the touch screen parses and restores the data packet to obtain the position coordinates of the identified object relative to the touch screen. The present invention can realize the positioning and object recognition of any object on a capacitive touch screen, and report the position of the object relative to the touch screen in real time, which can effectively solve the problem of low recognition accuracy when identifying different objects and moving objects, and the positioning is more accurate and flexible, which brings great expansibility to human-computer interaction facilities based on touch screens.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a touch screen object positioning method and system based on artificial intelligence. Background Art

[0002] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial Intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems.

[0003] With the continuous advancement of science and technology and artificial intelligence, human-computer interaction has also been widely used in various entertainment, technology exhibitions, service halls and other places, such as game interactive tables in the waiting hall of a cinema. However, most of the existing technologies use capacitive touch screens for basic human-computer interaction. The principle of most existing touch screens (supporting multi-touch capacitive screens) and object interaction devices is to use specific object models, and different object models contain different touch points at the bottom. When the object model is placed on the touch screen, the touch points at the bottom of the object model will trigger a touch event on the screen. The shapes of each touch point are calculated to distinguish different object models and the position of the touch screen where the object model is located, which is not conducive to the expansion of categories and requires the objects to be modified with touch points in advance. Therefore, it lacks a certain degree of fun and has poor flexibility. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a touch screen object positioning method and device based on artificial intelligence. Specifically, the embodiments of the present invention provide the following technical solutions:

[0005] In a first aspect, an embodiment of the present invention provides a touch screen object positioning method based on artificial intelligence, comprising:

[0006] Acquire an original image that at least completely covers the touch screen, and extract the touch screen from it to obtain a screen image to be identified;

[0007] Inputting the screen image to be identified into a deep learning network model to obtain position information of the object to be identified on the screen image to be identified;

[0008] generating a data packet based on the position information of the identified object, and sending the data packet to the touch screen;

[0009] The touch screen parses and restores the data packet to obtain the position coordinates of the identified object relative to the touch screen.

[0010] Furthermore, the obtaining of the original image that at least completely covers the touch screen, and extracting the touch screen to obtain the screen image to be identified includes:

[0011] Shooting the touch screen with a camera to obtain an original image;

[0012] Detecting the original image to obtain the coordinates of each endpoint of the touch screen contained in the original image;

[0013] The original image is processed based on the coordinates of each endpoint of the touch screen to obtain a screen image to be recognized.

[0014] Furthermore, the detecting the original image and obtaining the coordinates of each endpoint of the touch screen contained in the original image includes:

[0015] Performing grayscale conversion and median filtering on the original image to obtain image data;

[0016] Binarizing the image data to obtain a binary image;

[0017] The binary image is subjected to boundary recognition based on a boundary detection algorithm to obtain the coordinates of each endpoint of the touch screen contained in the original image.

[0018] Furthermore, the performing image processing on the original image based on the coordinates of each endpoint of the touch screen to obtain the screen image to be recognized includes:

[0019] Performing image cropping processing on the touch screen picture based on the coordinates of each endpoint of the touch screen to obtain a screen image;

[0020] The screen image is subjected to coordinate matrix transformation, and the screen image is scaled to a scaling multiple of the screen resolution to obtain the screen image to be identified.

[0021] Furthermore, the step of inputting the screen image to be identified into a deep learning network model to obtain position information of an object to be identified on the screen image to be identified includes:

[0022] Use deep learning network and target detection algorithm to build a deep learning target detection network;

[0023] Acquire a plurality of screen image sample data and corresponding annotation files to construct a data set, wherein the annotation files include location information and label names of identified objects in the screen image sample data;

[0024] Inputting the data set into the deep learning target detection network to generate the deep learning network model;

[0025] The screen image to be identified is input into the deep learning network model to obtain the position information and label name of the object to be identified on the screen image to be identified.

[0026] Furthermore, the position information of the object to be identified on the screen image to be identified includes: the position coordinates of the object to be identified relative to the position percentage coordinates of the screen image to be identified;

[0027] The touch screen parses and restores the data packet to obtain the position coordinates of the identified object relative to the touch screen, including: the touch screen parses and restores the data packet according to the position percentage coordinates to obtain the position coordinates of the identified object relative to the touch screen.

[0028] In a second aspect, the present invention further provides a touch screen object positioning device based on artificial intelligence, comprising:

[0029] Screen image extraction module: obtains the original image that at least completely covers the touch screen, and extracts the touch screen from it to obtain the screen image to be identified;

[0030] Position recognition module: used to input the screen image to be recognized into the deep learning network model to obtain the position information of the object to be recognized on the screen image to be recognized;

[0031] Sending module: generating a data packet based on the position information of the identified object, and sending the data packet to the positioning module;

[0032] Positioning module: used to parse and restore the data packet to obtain the position coordinates of the identified object relative to the touch screen.

[0033] In a third aspect, the present invention also provides an artificial intelligence-based touch screen object positioning system, comprising: an Ethernet camera, a touch screen and an AI host; wherein the Ethernet camera is placed parallel to and directly above the touch screen; and the AI ​​host comprises the device as described in the second aspect above.

[0034] In a fourth aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the touch screen object positioning method based on artificial intelligence as described in the first aspect are implemented.

[0035] In a fifth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the touch screen object positioning method based on artificial intelligence as described in the first aspect.

[0036] It can be seen from the above technical scheme that the touch screen object positioning method based on artificial intelligence provided by the present invention can realize the positioning and object recognition of any object on the capacitive touch screen and report the position of the object relative to the touch screen in real time by extracting the touch screen screen and inputting the screen image into the deep learning network model to obtain the position information of the identified object on the screen image, and the touch screen program analyzes the position information to obtain the position coordinates of the identified object. It can effectively solve the problem of low recognition accuracy when identifying different objects and moving objects, and the positioning is more accurate and flexible, which brings great scalability to human-computer interaction facilities based on touch screens. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0038] Figure 1 A schematic flow chart of a touch screen object positioning method based on artificial intelligence provided by an embodiment of the present invention.

[0039] Figure 2 A schematic diagram of a grayscale image of a touch screen object positioning method based on artificial intelligence provided by an embodiment of the present invention.

[0040] Figure 3 A schematic diagram of a binary image of a touch screen object positioning method based on artificial intelligence provided by an embodiment of the present invention.

[0041] Figure 4 A schematic diagram of boundary recognition of a touch screen object positioning method based on artificial intelligence provided by an embodiment of the present invention.

[0042] Figure 5 A schematic diagram of screen endpoint coordinates of a touch screen object positioning method based on artificial intelligence provided by an embodiment of the present invention.

[0043] Figure 6 A schematic diagram of the architecture of an artificial intelligence-based touch screen object positioning device provided by one embodiment of the present invention.

[0044] Figure 7 A schematic diagram of an artificial intelligence-based touch screen object positioning system provided by an embodiment of the present invention.

[0045] Figure 8 A flowchart of a touch screen object positioning system based on artificial intelligence provided by one embodiment of the present invention.

[0046] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0048] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] Figure 1 FIG. 1 is a flow chart of a touch screen object positioning method based on artificial intelligence provided by an embodiment of the present invention. Figure 1 As shown, the touch screen object positioning method based on artificial intelligence provided by the embodiment of the present invention includes the following steps:

[0050] Step S1: obtaining an original image that at least completely covers the touch screen, and extracting the touch screen from it to obtain a screen image to be identified.

[0051] Specifically, the original image can be obtained in the following ways: first, place an image acquisition tool such as a camera above the screen, and its acquisition range needs to be able to completely cover the touch screen to obtain the image data collected by the camera; secondly, capture a frame of image from the acquired video stream data for image analysis. Because the video stream data contains dynamic images, there will be blurred frames. Therefore, it is necessary to identify whether the current frame is a blurred frame image. If it is a blurred frame image, it is discarded. If it is a clear frame image, it is retained. The clear frame image is the original image. After that, the image of the complete touch screen screen is extracted from the clear frame image through image recognition, and then the image of the touch screen screen is cropped, stretched, compressed and other image processing is performed to obtain the screen image to be recognized. Among them, the screen image to be recognized refers to the image obtained after the above-mentioned image processing and other operations, which can be directly used by the artificial intelligence system to recognize objects on the screen.

[0052] In one embodiment of the present invention, the step of obtaining the original image that at least completely covers the touch screen and extracting the touch screen to obtain the screen image to be identified includes: step S11: photographing the touch screen with a camera to obtain the original image; step S12: detecting the original image to obtain the coordinates of each endpoint of the touch screen contained in the original image; step S13: performing image processing on the original image based on the coordinates of each endpoint of the touch screen to obtain the screen image to be identified.

[0053] Specifically, by photographing the touch screen with a camera, an original image covering the touch screen can be obtained. In addition to the complete touch screen, the original image also includes other objects that do not need to be located and identified, such as the edge of the touch screen. Therefore, it is necessary to detect the original image, obtain the coordinates of each endpoint of the touch screen contained in the original image, and obtain the image of the touch screen through the endpoint coordinates to eliminate the interference of other objects outside the screen. Furthermore, based on the coordinates of each endpoint of the touch screen, the original image is subjected to image processing operations such as cropping, stretching, and compression to obtain the screen image to be identified.

[0054] In one embodiment of the present invention, detecting the original image in step S12 and obtaining the coordinates of each endpoint of the touch screen contained in the original image includes: performing grayscale conversion and median filtering on the original image to obtain image data; binarizing the image data to obtain a binary picture; performing boundary recognition on the binary picture based on a boundary detection algorithm to obtain the coordinates of each endpoint of the touch screen contained in the original image.

[0055] Specifically, the original image is a clear frame image containing the entire touch screen obtained by directly photographing the touch screen with a camera. The clear frame image is converted into a grayscale image. The obtained grayscale image is as follows: Figure 2 At the same time, the image is processed by median filtering. The so-called median filtering refers to the processing technology of image filtering, which takes an intermediate value between the grayscale value of the image pixel and the grayscale value of the adjacent pixel, thereby suppressing the image noise.

[0056] The filtered image data is then binarized and converted into a binary image. The so-called binarization is an image segmentation method, and binarization can convert a grayscale image into a binary image. By setting a threshold, the grayscale of pixels greater than a certain critical grayscale value (i.e., the threshold) is set to the grayscale maximum value 255, and the grayscale of pixels less than this value (i.e., the threshold) is set to the grayscale minimum value 0, thereby achieving binarization, so that the entire image presents an obvious black and white effect, and a binary image is obtained. In this step, a threshold is set according to the scene, for example, the threshold is set to 180, and the image is binarized. By setting the threshold, a binary image in which the screen is white and the surroundings are black is obtained. The obtained image is as shown in FIG. Figure 3 As shown, the binarized image can effectively improve the efficiency of subsequent edge contour detection.

[0057] Further, in obtaining Figure 3After the binary image is binarized as shown, the boundary of the binary image can be identified based on the boundary detection algorithm. The boundary detection algorithm may include the machine vision Canny detection algorithm and the convex hull detection algorithm, but is not limited to the above two algorithms. Specifically, the boundary of the binary image is identified, and the machine vision Canny detection algorithm is used to identify the edge in the image and find the contour points in the image. The search result is as follows: Figure 4 As shown, by detecting the contour point convex hull detection algorithm, the four corner coordinates of the screen can be effectively detected, such as Figure 5 As shown, the coordinates of each endpoint of the touch screen contained in the original image can be obtained.

[0058] In one embodiment of the present invention, step S13 performs image processing on the original image based on the coordinates of each endpoint of the touch screen to obtain the screen image to be identified, including: performing image cropping processing on the touch screen picture based on the coordinates of each endpoint of the touch screen to obtain the screen image; performing a coordinate matrix transformation on the screen image, scaling the screen image to a scaling multiple of the screen resolution, to obtain the screen image to be identified.

[0059] Specifically, the touch screen image can be cropped according to the coordinates of each endpoint of the touch screen obtained in step S12 to obtain a touch screen screen image. The newly cropped screen image is transformed into a coordinate matrix and stretched and restored to a multiple of the original screen resolution or size (1920*1080). For example, the image is restored to a resolution or size of 480*270, and the scaling factor is 4. By scaling the screen image to a multiple of the screen resolution to obtain a screen image to be identified, the size of the image can be reduced, the processing of the deep learning algorithm can be facilitated, the speed of the algorithm calculation can be increased, and the efficiency of object positioning and identification can be improved.

[0060] Step S2: Input the screen image to be identified into the deep learning network model to obtain the position information of the object to be identified on the screen image to be identified.

[0061] Specifically, the screen image to be identified obtained in step S1 is input into the deep learning network model for target detection. The deep learning network model uses the Darknet deep learning network and the YoLo V3 target detection algorithm. Before deep learning reasoning, the deep learning network model needs to be trained to obtain a trained model file, and then reasoning is performed based on the model file to obtain the position information of the identified object, thereby realizing target detection based on the deep learning network model.

[0062] In one embodiment of the present invention, the step of inputting the screen image to be identified into the deep learning network model to obtain the position information of the object to be identified on the screen image to be identified includes: constructing a deep learning target detection network using a deep learning network and a target detection algorithm; obtaining multiple screen image sample data and corresponding annotation files to construct a data set, wherein the annotation file includes the position information and label name of the object to be identified in the screen image sample data; inputting the data set into the deep learning target detection network to generate a deep learning network model; inputting the screen image to be identified into the deep learning network model to obtain the position information and label name of the object to be identified on the screen image to be identified.

[0063] Specifically, in this step, the Darknet deep learning network and YoLo V3 target detection algorithm are used to construct a deep learning target detection network. First, the screen image sample data is obtained. Each image sample corresponds to a labeling file. The labeling file contains the position information, label name, etc. of the labeled object in the image sample, that is, the identified object. Secondly, a data set is constructed through multiple screen image sample data and their corresponding labeling files. The prepared data set is input into the deep learning target detection network, and a deep learning network model will be generated. Finally, the screen image to be identified is input into the model, and the input image is inferred according to the model file to obtain the position information and label name of the identified object on the screen image to be identified.

[0064] The embodiments of the present invention construct a deep learning network model, input the screen image into the deep learning network model, and obtain the position information of the identified object on the screen image. It can realize the positioning and object recognition of any object on the touch screen, and report the object position in real time, making the positioning more accurate and flexible.

[0065] In one embodiment of the present invention, the position information of the object to be identified on the screen image to be identified in this step includes: the position coordinates of the object to be identified in percentage relative to the position coordinates of the screen image to be identified.

[0066] The position percentage coordinates include the length of the position coordinates of the object to be identified in the x direction from the coordinate origin as a percentage of the horizontal length of the screen image to be identified and the length of the position coordinates of the object to be identified in the y direction from the coordinate origin as a percentage of the vertical length of the screen image to be identified. For example, the lower left corner of the image is taken as the coordinate origin (0, 0), the image size is 480*270, and the position coordinates of the object to be identified are (48, 54). Then, the length of the position coordinates of the object to be identified in the x direction from the coordinate origin is 48, which is 10% of the horizontal length 480 of the screen image to be identified; the length of the position coordinates of the object to be identified in the y direction from the coordinate origin is 54, which is 20% of the vertical length 270 of the screen image to be identified. The percentage position coordinates of the object to be identified relative to the coordinate origin are recorded as (10%, 20%), which represent the position information of the object to be identified on the screen image to be identified.

[0067] Step S3: Generate a data packet based on the position information of the identified object, and send the data packet to the touch screen.

[0068] Specifically, through the deep learning neural network model in step S2, the position information and label name of the object to be identified on the screen image to be identified can be obtained, a data packet is generated from the information, and the data packet is forwarded to the touch screen program through the UDP protocol.

[0069] Step S4: The touch screen parses and restores the data packet to obtain the position coordinates of the identified object relative to the touch screen.

[0070] Specifically, the touch screen program receives the data packet sent in step S3, and parses and restores the data packet to obtain the position information and label name of the object to be identified on the screen image to be identified. Based on the position information, the position coordinates of the object to be identified relative to the touch screen can be restored.

[0071] In one embodiment of the present invention, the touch screen in this step parses and restores the data packet to obtain the position coordinates of the identified object relative to the touch screen, including: the touch screen parses and restores the data packet according to the position percentage coordinates to obtain the position coordinates of the identified object relative to the touch screen.

[0072] Specifically, the position percentage coordinates represent the percentage of the length of the position coordinates of the object to be identified in the x direction from the coordinate origin to the horizontal length of the screen image to be identified and the percentage of the length of the position coordinates of the object to be identified in the y direction from the coordinate origin to the vertical length of the screen image to be identified. For example, since the size of the screen image to be identified is scaled, its size is 480*270, and the size of the touch screen is 1920*1080, the lower left corner of the image and the screen is used as the coordinate origin, the position coordinates of the object to be identified relative to the screen image to be identified are (48, 54), the percentage position coordinates are (10%, 20%), and the position of the object to be identified is restored to the touch screen according to the percentage position coordinates, and the horizontal coordinate of the object relative to the touch screen is 1920*10%=192, and the vertical coordinate is 1080*20%=216, so the position coordinates of the object to be identified relative to the touch screen are (192, 216).

[0073] On the basis of the above embodiments, the present invention extracts the touch screen screen and inputs the screen image into the deep learning network model to obtain the position information of the identified object on the screen image, and the touch screen program parses the position information to obtain the position coordinates of the identified object, which can realize the positioning and object recognition of any object on the capacitive touch screen, and report the position of the object relative to the touch screen in real time. It can effectively solve the problem of low recognition accuracy when identifying different objects and moving objects, and the positioning is more accurate and flexible, which brings great scalability to human-computer interaction facilities based on touch screens.

[0074] Corresponding to the above embodiment, another embodiment of the present invention further provides a touch screen object positioning device based on artificial intelligence. In this embodiment, for the contents that are the same or corresponding to the above embodiment, please refer to the above introduction and will not be repeated later.

[0075] like Figure 6 As shown, an artificial intelligence-based touch screen object positioning device disclosed in an embodiment of the present invention includes a screen image extraction module 601, a position recognition module 602, a sending module 603, and a positioning module 604, wherein:

[0076] Screen image extraction module 601: obtaining an original image that at least completely covers the touch screen, and extracting the touch screen from it to obtain a screen image to be identified, including: photographing the touch screen through a camera to obtain an original image; detecting the original image to obtain the coordinates of each endpoint of the touch screen contained in the original image; performing image processing on the original image based on the coordinates of each endpoint of the touch screen to obtain a screen image to be identified; the detecting of the original image to obtain the coordinates of each endpoint of the touch screen contained in the original image includes: performing grayscale conversion and median filtering on the original image to obtain image data; binarizing the image data to obtain a binarized image; performing boundary recognition on the binarized image based on a boundary detection algorithm to obtain the coordinates of each endpoint of the touch screen contained in the original image; the image processing on the original image based on the coordinates of each endpoint of the touch screen to obtain the screen image to be identified includes: performing image cropping on the touch screen image based on the coordinates of each endpoint of the touch screen to obtain a screen image; performing coordinate matrix transformation on the screen image to scale the screen image to a scaling multiple of the screen resolution to obtain the screen image to be identified;

[0077] Position identification module 602: used to input the screen image to be identified into the deep learning network model to obtain the position information of the identified object on the screen image to be identified, including: using the deep learning network and the target detection algorithm to construct a deep learning target detection network; acquiring multiple screen image sample data and corresponding annotation files to construct a data set, wherein the annotation file includes the position information and label name of the identified object in the screen image sample data; inputting the data set into the deep learning target detection network to generate the deep learning network model; inputting the screen image to be identified into the deep learning network model to obtain the position information and label name of the identified object on the screen image to be identified; the position information of the identified object on the screen image to be identified includes: the position coordinates of the identified object relative to the position percentage coordinates of the screen image to be identified;

[0078] Sending module 603: generating a data packet based on the position information of the identified object, and sending the data packet to the positioning module;

[0079] Positioning module 604: used to parse and restore the data packet to obtain the position coordinates of the identified object relative to the touch screen; the touch screen parses and restores the data packet to obtain the position coordinates of the identified object relative to the touch screen, including: the touch screen parses and restores the data packet according to the position percentage coordinates to obtain the position coordinates of the identified object relative to the touch screen.

[0080] The touch screen object positioning device based on artificial intelligence provided by the embodiment of the present invention adopts the touch screen object positioning method based on artificial intelligence in the above-mentioned embodiment 1, obtains the position information of the identified object on the screen image by extracting the touch screen screen and inputting the screen image into the deep learning network model, and the touch screen program parses the position information to obtain the position coordinates of the identified object, which can realize the positioning and object recognition of any object on the capacitive touch screen, and report the position of the object relative to the touch screen in real time. It can effectively solve the problem of low recognition accuracy when identifying different objects and moving objects, and the positioning is more accurate and flexible, which brings great scalability to human-computer interaction facilities based on touch screens.

[0081] Since the artificial intelligence-based touch screen object positioning device provided in the embodiment of the present invention can be used to execute the artificial intelligence-based touch screen object positioning method described in the above embodiment, its working principle and beneficial effects are similar, so they are not described in detail here. For specific contents, please refer to the introduction of the above embodiment.

[0082] Another embodiment of the present invention also provides a touch screen object positioning system based on artificial intelligence, such as Figure 7 As shown, it includes: a network camera 701, a touch screen 702 and an AI host 703; wherein the network camera 701 is placed parallel to and directly above the touch screen 702; the AI ​​host 703 includes a touch screen object positioning device based on artificial intelligence as in the above-mentioned embodiment. The workflow diagram of the system is shown in FIG. Figure 8 shown.

[0083] Since the artificial intelligence-based touch screen object positioning system provided in the embodiment of the present invention can be used to execute the artificial intelligence-based touch screen object positioning method described in the above embodiment, its working principle and beneficial effects are similar, so they will not be described in detail here. For specific contents, please refer to the introduction of the above embodiment.

[0084] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, see Fig. 9, the electronic device specifically includes the following contents: a processor, a memory, a communication interface and a communication bus; wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the processor is used to call the computer program in the memory, and the processor implements all the steps of the above method when executing the computer program. For example, when the processor executes the computer program, the following process is implemented: obtaining an original image that at least completely covers the touch screen, and extracting the touch screen from it to obtain a screen image to be identified; inputting the screen image to be identified into a deep learning network model to obtain the position information of the object to be identified on the screen image to be identified; generating a data packet based on the position information of the object to be identified, and sending the data packet to the touch screen; the touch screen parses and restores the data packet to obtain the position coordinates of the object to be identified relative to the touch screen.

[0085] It can be understood that the detailed functions and extended functions that can be executed by the computer program can refer to the description of the above embodiments.

[0086] Based on the same inventive concept, another embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, all the steps of the above method are implemented. For example, when the processor executes the computer program, the following process is implemented: obtaining an original image that at least completely covers the touch screen, and extracting the touch screen from it to obtain a screen image to be identified; inputting the screen image to be identified into a deep learning network model to obtain position information of the object to be identified on the screen image to be identified; generating a data packet based on the position information of the object to be identified, and sending the data packet to the touch screen; the touch screen parses and restores the data packet to obtain the position coordinates of the object to be identified relative to the touch screen.

[0087] It can be understood that the detailed functions and extended functions that can be executed by the computer program can refer to the description of the above embodiments.

[0088] Based on the same inventive concept, another embodiment of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements all the steps of the above method. For example, when the processor executes the computer program, it implements the following process: acquiring an original image that at least completely covers the touch screen, and extracting the touch screen from it to obtain a screen image to be identified; inputting the screen image to be identified into a deep learning network model to obtain position information of an object to be identified on the screen image to be identified; generating a data packet based on the position information of the object to be identified, and sending the data packet to the touch screen; the touch screen parses and restores the data packet to obtain the position coordinates of the object to be identified relative to the touch screen.

[0089] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0090] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present invention. Those of ordinary skill in the art may understand and implement them without creative effort.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the traffic audit method described in each embodiment or some parts of the embodiment.

[0092] In addition, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0093] In addition, in the present invention, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A touch screen object positioning method based on artificial intelligence, characterized in that: include: Acquire an original image that at least completely covers the touch screen, and extract the touch screen from it to obtain a screen image to be identified; Inputting the screen image to be identified into a deep learning network model to obtain position information of the object to be identified on the screen image to be identified; generating a data packet based on the position information of the identified object, and sending the data packet to the touch screen; The touch screen parses and restores the data packet to obtain the position coordinates of the identified object relative to the touch screen; The step of inputting the screen image to be identified into a deep learning network model to obtain position information of an object to be identified on the screen image to be identified comprises: Use deep learning network and target detection algorithm to build a deep learning target detection network; Acquire a plurality of screen image sample data and corresponding annotation files to construct a data set, wherein the annotation files include location information and label names of identified objects in the screen image sample data; Inputting the data set into the deep learning target detection network to generate the deep learning network model; The screen image to be identified is input into the deep learning network model to obtain the position information and label name of the object to be identified on the screen image to be identified.

2. The touch screen object positioning method based on artificial intelligence according to claim 1 is characterized in that: The obtaining of the original image that at least completely covers the touch screen, and extracting the touch screen to obtain the screen image to be identified includes: Shooting the touch screen with a camera to obtain an original image; Detecting the original image to obtain the coordinates of each endpoint of the touch screen contained in the original image; The original image is processed based on the coordinates of each endpoint of the touch screen to obtain a screen image to be recognized.

3. The touch screen object positioning method based on artificial intelligence according to claim 2 is characterized in that: The detecting the original image and obtaining the coordinates of each endpoint of the touch screen contained in the original image comprises: Performing grayscale conversion and median filtering on the original image to obtain image data; Binarizing the image data to obtain a binary image; The binary image is subjected to boundary recognition based on a boundary detection algorithm to obtain the coordinates of each endpoint of the touch screen contained in the original image.

4. The touch screen object positioning method based on artificial intelligence according to claim 2 is characterized in that: The performing image processing on the original image based on the coordinates of each endpoint of the touch screen to obtain the screen image to be recognized includes: Performing image cropping processing on the touch screen picture based on the coordinates of each endpoint of the touch screen to obtain a screen image; The screen image is subjected to coordinate matrix transformation, and the screen image is scaled to a scaling multiple of the screen resolution to obtain the screen image to be identified.

5. The touch screen object positioning method based on artificial intelligence according to claim 1, characterized in that: The position information of the object to be identified on the screen image to be identified includes: the position coordinates of the object to be identified relative to the position percentage coordinates of the screen image to be identified; The touch screen parses and restores the data packet to obtain the position coordinates of the identified object relative to the touch screen, including: the touch screen parses and restores the data packet according to the position percentage coordinates to obtain the position coordinates of the identified object relative to the touch screen.

6. A touch screen object positioning device based on artificial intelligence, characterized in that: The device comprises: Screen image extraction module: obtains the original image that at least completely covers the touch screen, and extracts the touch screen from it to obtain the screen image to be identified; Position identification module: used for inputting the screen image to be identified into a deep learning network model to obtain the position information of the object to be identified on the screen image to be identified; the inputting the screen image to be identified into the deep learning network model to obtain the position information of the object to be identified on the screen image to be identified includes: constructing a deep learning target detection network using a deep learning network and a target detection algorithm; acquiring a plurality of screen image sample data and corresponding annotation files to construct a data set, wherein the annotation file includes the position information and label name of the identified object in the screen image sample data; inputting the data set into the deep learning target detection network to generate the deep learning network model; inputting the screen image to be identified into the deep learning network model to obtain the position information and label name of the object to be identified on the screen image to be identified; Sending module: generating a data packet based on the position information of the identified object, and sending the data packet to the positioning module; Positioning module: used to parse and restore the data packet to obtain the position coordinates of the identified object relative to the touch screen.

7. A touch screen object positioning system based on artificial intelligence, characterized in that: include: An internet camera, a touch screen and an AI host; wherein the internet camera is placed parallel to and directly above the touch screen; and the AI ​​host comprises the device as described in claim 6 above.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the touch screen object positioning method based on artificial intelligence are implemented as described in any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the touch screen object positioning method based on artificial intelligence as described in any one of claims 1 to 5 are implemented.

Images