Light spot positioning method, device and equipment and computer readable storage medium

Abstract

This application provides a method, apparatus, device, and computer-readable storage medium for locating a light spot. The method includes: acquiring at least one frame of an image obtained by an image acquisition device capturing images of the screen during content display; wherein the image includes a light spot generated by a light emitting device emitting light to perform an interactive operation on the content displayed on the screen; determining the graphic outline of the screen in the image, and acquiring a screen image corresponding to the screen based on the graphic outline; performing image correction on the screen image to obtain a frontal view image when the image acquisition device is looking directly at the screen; determining the position of the light spot in the frontal view image, and performing an interactive response to the interactive operation based on the position. This application can improve the accuracy of light spot location and increase the fun of interactive operations performed based on the light spot.

Description

Technical Field

[0001] This application relates to human-computer interaction and computer vision technology, and in particular to a method, apparatus, device, computer-readable storage medium, and computer program product for locating light spots. Background Technology

[0002] With the widespread use of laser pointers in multimedia presentation systems such as conferences, lectures, and teaching, speakers can use laser pointers to manipulate their presentations away from the computer, thus enabling them to interact with the audience more broadly and complete the task of information delivery.

[0003] In related technologies, although some projection devices support simple interactive operations such as page turning and confirmation via laser pointers, the low accuracy of obtaining the real-time coordinate position of the laser pointer's projected spot on the screen results in poor interactive effects when many applications or games interact with users via laser pointers. Summary of the Invention

[0004] This application provides a method, apparatus, device, computer-readable storage medium, and computer program product for locating light spots, which can improve the accuracy of light spot positioning and increase the fun of interactive operations performed based on light spots.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for locating a light spot, including:

[0007] During the display of content on the screen, at least one frame of image is obtained by the image acquisition device from the screen.

[0008] The image includes: a light spot generated by the light emitting device performing interactive operations on the content displayed on the screen by emitting light;

[0009] Determine the graphic outline of the screen in the image, and obtain the screen image corresponding to the screen based on the graphic outline of the screen;

[0010] The screen image is corrected to obtain a front view image corresponding to the image acquisition device viewing the screen and acquiring the image.

[0011] Determine the position of the light spot in the front view image, and based on the position, execute an interactive response for the interactive operation.

[0012] This application provides a light spot positioning device, comprising:

[0013] The acquisition module is used to acquire at least one frame of image obtained by the image acquisition device from the screen during the content display process; wherein, the image includes: a light spot generated by the light emitting device performing an interactive operation on the content displayed on the screen by emitting light;

[0014] The determining module is used to determine the graphic outline of the screen in the image, and to obtain the screen image corresponding to the screen based on the graphic outline of the screen;

[0015] The correction module is used to perform image correction on the screen image to obtain a front view image corresponding to the image acquisition device viewing the screen directly when acquiring the image;

[0016] The positioning module is used to determine the position of the light spot in the front view image, and based on the position, execute an interactive response for the interactive operation.

[0017] In the above scheme, the determining module is further used to perform image recognition on the image to obtain the region contours corresponding to each object entity included in the image;

[0018] Obtain the area of ​​the region outline corresponding to each object entity, and determine the region outline with the largest area as the target region outline, and the corresponding object entity as the screen.

[0019] The vertices on the contour of the target region are obtained, and the graphic contour of the screen in the image is determined based on the vertices.

[0020] In the above scheme, the determining module is further used to perform binarization processing on the image to obtain a binarized image;

[0021] Perform connectivity analysis on the binarized image to obtain the image regions corresponding to each object entity included in the image;

[0022] Extract the contour points of the edges of each of the image regions, and determine the region contours corresponding to each object entity included in the image based on the contour points of the edges of each of the image regions.

[0023] In the above scheme, the determining module is further used to determine the distance between adjacent vertices on the contour of the target region;

[0024] Based on the distance between adjacent vertices, multiple vertices included in the target region contour are filtered to obtain multiple target vertices;

[0025] By connecting the multiple target vertices, the initial outline of the screen is obtained, and the initial outline is smoothed to obtain the graphic outline of the screen in the image.

[0026] In the above scheme, the determining module is further used to perform binarization processing on the image to obtain a binarized image including at least two graphic contours, wherein the graphic contours include multiple line segments;

[0027] Multiple line segments are filtered to determine target line segments for constructing the graphic outline of the screen, and the graphic outline of the screen in the image is determined based on the target line segments.

[0028] In the above scheme, the determining module is further used to determine the length of each line segment, and to construct a coordinate system using the image as the front view of the image acquisition device, and to obtain the slope of each line segment in the coordinate system;

[0029] When the length of the line segment reaches the length threshold, the multiple line segments are classified based on the slope to obtain the horizontal line segments and vertical line segments corresponding to the coordinate system.

[0030] Based on the horizontal and vertical line segments, target line segments are determined for constructing the graphic outline of the screen.

[0031] In the above scheme, the determining module is further configured to determine an initial graphic contour corresponding to the screen from at least two graphic contours included in the binarized image;

[0032] A coordinate system is constructed using the image as the front view of the image acquisition device, and the horizontal and vertical central axes of the initial graphic contour are determined in the coordinate system.

[0033] Based on the position of each line segment relative to the horizontal central axis, the line segments are grouped to obtain a first horizontal line segment located above the horizontal central axis and a second horizontal line segment located below the horizontal central axis.

[0034] Based on the position of each line segment relative to the longitudinal central axis, the line segments are grouped to obtain a first longitudinal line segment located to the left of the longitudinal central axis and a second longitudinal line segment located to the right of the longitudinal central axis.

[0035] Based on the first horizontal line segment, the second horizontal line segment, the first vertical line segment, and the second vertical line segment, a target line segment corresponding to the graphic outline used to construct the screen is determined.

[0036] In the above scheme, the determining module is further configured to, when filtering multiple line segments and no target line segment for constructing the graphic contour of the screen is obtained, construct a coordinate system using the image as the front view of the image acquisition device, and determine the horizontal center axis and vertical center axis of each graphic contour in the binary image in the coordinate system.

[0037] Based on each of the said horizontal central axes, the first horizontal sub-contour and the second horizontal sub-contour of each of the said graphic contours are determined;

[0038] Based on each of the longitudinal center axes, the first longitudinal sub-contour and the second longitudinal sub-contour of each of the graphic contours are determined.

[0039] Based on the first horizontal sub-contour, the second horizontal sub-contour, the first vertical sub-contour, and the second vertical sub-contour, the graphic contour of the screen in the image is determined.

[0040] In the above scheme, the correction module is further used to obtain the first coordinate system corresponding to the image acquisition device when the image is used as the front view of the image acquisition device, and the second coordinate system corresponding to the screen when the image acquisition device is looking at the screen and performing image acquisition.

[0041] Based on the first coordinate system and the second coordinate system, coordinate transformation is performed on each pixel in the screen image to obtain a front view image corresponding to the image acquisition device viewing the screen and acquiring the image.

[0042] In the above scheme, when the image includes at least two light spots, the positioning module is further used to obtain the value of the light spot parameter of each light spot, wherein the light spot parameter includes at least one of the light spot shape and the light spot brightness;

[0043] Based on the light spot parameters, the light spot with the largest value of the light spot parameters is selected as the target light spot, and the position of the target light spot in the front view image is determined.

[0044] In the above scheme, when the number of images is multiple frames, the positioning module is further used to obtain the position information of the light spot in each of the images;

[0045] Based on the location information, the trajectory of the light spot is determined;

[0046] The spot on the running trajectory is taken as the target laser spot, and the position of the target laser spot in the front view image is determined.

[0047] In the above scheme, the positioning module is also used to perform dot-shaped drawing corresponding to the light spot at the position when the interactive operation is a graphics drawing operation;

[0048] When the interaction operation is a trigger operation for the content, the trigger result of the trigger operation corresponding to the content is output.

[0049] This application provides an electronic device, including:

[0050] Memory, used to store executable instructions;

[0051] The processor, when executing executable instructions stored in the memory, implements the spot positioning method provided in the embodiments of this application.

[0052] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the spot positioning method provided in this application.

[0053] This application provides a computer program product, including a computer program or instructions, for inducing a processor to execute and implement the spot positioning method provided in this application.

[0054] The embodiments of this application have the following beneficial effects:

[0055] By applying the embodiments of this application, the graphic outline of the screen in the image including the screen and the light spot acquired by the image acquisition device is determined, the corresponding screen image is determined, and the screen image is corrected to obtain a front view image when the image acquisition device is looking directly at the screen to acquire the image. The target position of the light spot is determined in the front view image. In this way, the position of the light spot can be accurately located in the screen image from various angles, improving the accuracy of the light spot positioning. Finally, based on the position of the light spot, an interactive response for the interactive operation is executed, thus increasing the fun of the interactive operation based on the light spot. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the architecture of the spot positioning system provided in the embodiments of this application;

[0057] Figure 2 This is a schematic diagram of the structure of an electronic device implementing the spot positioning method provided in the embodiments of this application;

[0058] Figure 3 This is a schematic flowchart of the spot positioning method provided in the embodiments of this application;

[0059] Figure 4 This is a flowchart illustrating the method for determining a graphic contour provided in an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of the region outline provided in an embodiment of this application;

[0061] Figure 6 This is a flowchart illustrating the method for determining a graphic contour provided in an embodiment of this application;

[0062] Figure 7 This is a flowchart illustrating the method for creating a graphical outline of a screen provided in an embodiment of this application.

[0063] Figure 8This is a flowchart illustrating the smoothing process method provided in the embodiments of this application;

[0064] Figure 9 This is a schematic diagram illustrating the method for determining the target line segment provided in an embodiment of this application;

[0065] Figure 10 This is a schematic diagram illustrating the method for determining the graphic outline of a screen according to an embodiment of this application;

[0066] Figure 11 This is a schematic diagram of a line segment method for determining the graphic outline of a screen, provided in an embodiment of this application.

[0067] Figure 12 This is a schematic diagram of the method for determining the graphic outline of a screen provided in an embodiment of this application;

[0068] Figures 13A-13B This is a schematic diagram of the light spot-based interactive operation provided in an embodiment of this application;

[0069] Figure 14 These are interactive diagrams of game applications provided by related technologies;

[0070] Figure 15 This is a schematic diagram of the interactive operation of a light-emitting device provided in an embodiment of this application;

[0071] Figure 16A This is a schematic diagram of the complete process for locating the light spot provided in the embodiments of this application;

[0072] Figure 16B This is a visual schematic diagram of the spot positioning method provided in the embodiments of this application;

[0073] Figure 17 This is a schematic diagram of the spot positioning method provided in the embodiments of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0075] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0076] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0077] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0079] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0080] 1) Grayscale image: Also known as a gray-scale image, each pixel in an image typically corresponds to a specific location in two-dimensional space, and has one or more sampled values ​​associated with that point. Each pixel in the image can be represented by a brightness value (intensity) from 0 (black) to 255 (white). 0-255 represents different grayscale levels.

[0081] 2) Binary image: refers to a grayscale image in which there are only two gray levels. That is to say, the gray value of any pixel in the image is 0 or 255, representing black and white respectively.

[0082] 3) Mean Filtering: Mean filtering is a typical linear filtering algorithm. It involves applying a template to the target pixel in the image, which includes its surrounding eight pixels (the eight pixels surrounding the target pixel form a filter template, including the target pixel itself). The original pixel value is then replaced by the average value of all pixels in the template. In other words, the pixel value of each point is replaced by the average value of its eight surrounding pixels. Mean filtering is mainly used to smooth images. In practical applications, some images have very high sharpness; using this mean filtering algorithm can reduce the sharpness, making the image look more natural.

[0083] Based on the above explanation of the nouns and terms used in the embodiments of this application, the spot positioning system provided in the embodiments of this application is described below. See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of a spot positioning system provided in an embodiment of this application. To support the positioning application of a single spot, in the spot positioning system 100, the terminal 400 connects to the server 200 via a network 300. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. The server 200 can belong to a target server cluster, which includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server cluster can be used to provide backend services for applications supporting a 3D virtual environment.

[0084] Terminal 400 is used to install and run target application client 410 containing the target application. In response to the user's startup operation for the target application, it runs the target application and displays the relevant content of the target application.

[0085] Image acquisition device 420 is used to acquire images of the screen during the display of content of a target application on the screen of a terminal, capture at least one frame of the screen image, and provide an application programming interface for other devices to access the captured at least one frame of the image.

[0086] Server 200 is used to acquire at least one frame of image obtained by an image acquisition device from the screen during the content display process. The image includes a light spot generated by a light emitting device emitting light to perform interactive operations on the content displayed on the screen. Server 200 is used to determine the graphic outline of the screen in the image and acquire the corresponding screen image based on the graphic outline of the screen. Server 200 is used to perform image correction on the screen image to obtain a front view image when the image acquisition device is looking directly at the screen. Server 200 is used to determine the position of the light spot in the front view image and execute an interactive response for the interactive operation based on the position.

[0087] In practical applications, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smart TV, smartwatch, etc., but is not limited to these. Image acquisition device 410 can be an independent infrared camera, an independent ordinary camera, a camera built into the terminal, etc., but is not limited to these. Light emitting device can be a laser pointer, a virtual reality (VR) controller, a VR ring, etc., but is not limited to these. Terminal 400 and server 200 can be directly or indirectly connected via wired or wireless communication, and image acquisition device 410 and server 200 can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions on this connection.

[0088] The embodiments of this application can also be implemented with the help of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the calculation, storage, processing, and sharing of data.

[0089] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0090] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device implementing the spot positioning method provided in this application embodiment. In practical applications, the electronic device 500 can be implemented as follows: Figure 1 The server or terminal in the application will be used to describe the electronic device that implements the spot positioning method of the embodiments of this application. Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2The general labeled all buses as Bus System 540.

[0091] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0092] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0093] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0094] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0095] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0096] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0097] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0098] Presentation module 553 is used to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with user interface 530.

[0099] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0100] In some embodiments, the spot positioning device provided in this application can be implemented in software. Figure 2 The diagram illustrates the structure of an electronic device provided in this application as a server for locating light spots. The light spot locating device 555, stored in memory 550, can be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 5551, a determination module 5552, a correction module 5553, and a positioning module 5554. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0101] In other embodiments, the spot positioning device provided in this application can be implemented in hardware. As an example, the spot positioning device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the spot positioning method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0102] Based on the above description of the spot positioning system and electronic device provided in the embodiments of this application, the spot positioning method provided in the embodiments of this application is described below. In some embodiments, the spot positioning method provided in the embodiments of this application can be implemented by a server or a terminal alone, or by a server and a terminal working together. In some embodiments, the terminal or server can implement the spot positioning method provided in the embodiments of this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a client that supports virtual scenes, such as a game APP; it can also be a mini-program, that is, a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0103] The following uses a server implementation as an example to illustrate the spot positioning method provided in this application. See also: Figure 3 , Figure 3 This is a flowchart illustrating the spot positioning method provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained.

[0104] In step 101, during the process of displaying content on the screen, the server acquires at least one frame of image obtained by the image acquisition device from the screen.

[0105] It should be noted that the captured images include: light spots generated by the light emitting device performing interactive operations on the content displayed on the screen by emitting light.

[0106] In practical implementation, the screen includes, but is not limited to, projection screens and LCD displays. The content displayed on the screen is the content of the target application, such as a game application or an office application. The server can be a backend server processing the current target application, or a separately deployed server for image processing. After obtaining the processing results of the image, it provides an application programming interface (API) for the target application to call. In practical applications, the image acquisition device (such as a regular camera) can be placed arbitrarily, so that at least one frame of the image captured on the screen can be from any angle. In addition, users can interact with the content of the target application displayed on the screen by controlling a light emitting device to project a light spot (such as a laser pointer, VR (Virtual Reality) controller, VR ring, etc.). It should be noted that the light here refers to light within the visible spectrum, such as ordinary laser emitted by a regular laser device or infrared laser emitted by an infrared laser device. The server can receive at least one frame of the image captured on the screen by the image acquisition device, and the image includes the light spot projected by the light emitting device and the screen itself.

[0107] In step 102, the graphic outline of the screen in the image is determined, and the screen image corresponding to the screen is obtained based on the graphic outline of the screen.

[0108] In practical implementation, the server can perform image preprocessing operations on the color images acquired by the image acquisition device. Image preprocessing operations can include image grayscale conversion and image binarization. Image grayscale conversion refers to processing an image to obtain a grayscale image. Like color images, grayscale images can reflect the overall and local distribution and characteristics of color and brightness levels in the entire image; however, processing grayscale images can effectively reduce the amount of computation. In practical applications, methods for image grayscale conversion include component method, maximum value method, average value method, and weighted average method. Among these, the average value method involves calculating a simple average of the brightness of the three components (R (red), G (green), and B (blue)) in the color image, and outputting the resulting value as the grayscale value to obtain a grayscale image. Simultaneously, to highlight the outlines of the various objects (objects captured by the image acquisition device) included in the grayscale image, binarization processing can also be performed on the grayscale image to obtain a binary image.

[0109] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the method for determining the graphic contour provided in an embodiment of this application. Figure 3 Step 102 shown can be implemented through steps 1021a to 1023a, which will be explained in conjunction with each step.

[0110] Step 1021a: The server performs image recognition on the image to obtain the region contours corresponding to each object entity included in the image.

[0111] In practice, the server performs image recognition on images including light spots (this may include grayscale processing of the image to obtain a grayscale image, followed by binarization of the grayscale image to obtain a binarized image). Based on the binarized image, the contours of the regions corresponding to each object entity in the image can be determined.

[0112] For example, see Figure 5 , Figure 5 This is a schematic diagram of the region outline provided in an embodiment of this application. Figure 5 (1) shows a scenario where a user holds a light-emitting device (such as a laser pointer) and stands in front of a screen displaying a target application to project a laser beam. Figure 5 (2) Shows at least one frame of image of the screen captured by the camera and received by the server. The image includes the screen and the laser spot. Figure 5 (3) shows the server performing noise reduction on the image, that is, reducing the level of detail by blurring the image. Figure 5 (4) shows the binarized image obtained by binarizing the image. The binarized image can highlight the regional outlines of each object included in the image. In practical applications, the projection brightness of the screen is higher than the ambient brightness around the screen, allowing interference from other light sources such as lights and luminous objects.

[0113] In some embodiments, see Figure 6 , Figure 6 This is a flowchart illustrating the method for determining the graphic contour provided in an embodiment of this application. Figure 4 Step 1021a shown can be implemented through steps 201-203, which will be explained in conjunction with each step.

[0114] Step 201: The server performs binarization processing on the image to obtain a binarized image.

[0115] In practice, the server performs binarization processing on the image, that is, the color image captured by the image acquisition device is converted into grayscale to obtain a grayscale image, and then the grayscale image is binarized to obtain the corresponding binarized image. In this way, the outlines of each object in the image can be more prominently displayed.

[0116] For example, see Figure 5 (4) The diagram shows a binarized image. The outlines of objects in the image can be highlighted by black (i.e., setting the gray value of the pixel to 0) and white (i.e., setting the gray value of the pixel to 255).

[0117] Step 202: Perform connectivity analysis on the binarized image to obtain the image regions corresponding to each object entity included in the image.

[0118] In practice, the server scans the entire binarized image to find all connected color patches (i.e., to obtain the region defined by at least two consecutive pixels with a gray value of 255) and determines the contour points of each connected color patch. Since a connected color patch is a region that includes multiple consecutive pixels, the contour points of the connected color patches can be used to determine the image region corresponding to each object entity in the screen image.

[0119] For example, see Figure 5 (4) The multiple white areas shown are multiple connected color blocks in the binarized image. The image areas of each object entity in the screen image (such as the rectangular area at the bottom, the irregular graphic area at the top, and the long strip in the middle) are determined by each pixel point on the outline of each connected color block.

[0120] Step 203: Extract the contour points of the edges of each image region, and determine the region contours corresponding to each object entity included in the image based on the contour points of the edges of each image region.

[0121] In practice, the server extracts the contour points (i.e., pixels) of the image region edges corresponding to each object entity in the binarized image, and connects adjacent pixels to obtain the region contours corresponding to each object entity (such as the screen).

[0122] Step 1022a: Obtain the area of ​​the region outline corresponding to each object entity, and determine the region outline with the largest area as the target region outline, and the corresponding object entity as the screen.

[0123] In actual implementation, the server determines the area of ​​each region's outline, and the area can be calculated based on the number of pixels included within the region's outline.

[0124] For example, see Figure 5 (4) Show the outline of each region. It can be determined that the region outline of the large rectangle in the figure has the largest area. At this time, the outline of this region is determined as the target region outline (i.e. the region outline corresponding to the screen).

[0125] Step 1023a: Obtain the vertices on the contour of the target region, and determine the graphic contour of the screen in the image based on the vertices.

[0126] In actual implementation, after the server determines the outline of the target area where the screen is located in the binarized image, the server obtains the vertices on the current target area outline, and the server obtains the graphic outline of the screen in the image based on each vertex.

[0127] In some embodiments, see Figure 7 , Figure 7 This is a flowchart illustrating the screen graphic outline method provided in an embodiment of this application. Figure 4 Step 1023a shown can be implemented through steps 301 to 303, which will be explained in conjunction with each step.

[0128] Step 301: The server determines the distance between adjacent vertices on the outline of the target region.

[0129] In practice, the target region outline determined by the server includes multiple vertices (pixels), and each vertex is randomly distributed on both sides of a straight line. The server can filter the vertices that constitute the target region outline based on the distance between adjacent vertices. If the distance between adjacent vertices is too small, points that are too close to the current point can be directly excluded, and other vertices can be traversed.

[0130] For example, see Figure 5 (4) The large rectangle shown is the outline of the target area, that is, the outline of the area (rectangle) corresponding to the screen. There are multiple white pixels randomly distributed on the two sides of each side of the rectangle (these pixels can be used as vertices to construct the outline of the target area). When constructing the outline of the target area, the vertices to construct the outline of the target area can be selected from the randomly distributed pixels according to the distance between adjacent points.

[0131] Step 302: Based on the distance between adjacent vertices, filter the multiple vertices included in the target region contour to obtain multiple target vertices.

[0132] In practice, the server can select multiple target vertices from the multiple vertices included in the target region contour based on the distance between adjacent vertices, and use them to construct the initial contour of the screen in the image.

[0133] For example, see Figure 5 (4) 100 pixels are randomly distributed on both sides of the straight line of the large rectangular area outline. Based on the distance between adjacent points and the preset distance threshold between adjacent points when determining the target vertex, pixels with too small distance are excluded. Finally, 30 vertices are selected to construct the initial outline of the screen.

[0134] Step 303: Connect multiple target vertices to obtain the initial outline of the screen, and smooth the initial outline to obtain the graphic outline of the screen in the image.

[0135] In actual implementation, the server determines the initial outline of the screen based on a number of target vertices. Since the target vertices are randomly distributed and not along a straight line, the server can smooth the initial outline to obtain the graphic outline of the screen in the image in order to accurately determine the graphic outline of the screen.

[0136] In some embodiments, see Figure 8 , Figure 8 This is a flowchart illustrating the smoothing process provided in the embodiments of this application. Figure 3 Step 102 shown can also be implemented through steps 1021b to 1023b, which will be explained in conjunction with each step.

[0137] Step 1021b: The server performs binarization processing on the image to obtain a binarized image that includes at least two graphic contours.

[0138] In actual implementation, the server performs binarization processing on the images acquired by the image acquisition device to obtain a binarized image that includes multiple graphic contours. The graphic contours may include at least one of the following: the graphic contour corresponding to the screen, and the graphic contour corresponding to the interference object.

[0139] For example, see Figure 5 (4) The outline of the white area shown is the graphic outline of each object in the image captured by the image acquisition device, including the graphic outline of the screen captured.

[0140] Step 1022b: Construct multiple line segments based on the vertices on each graphic contour.

[0141] In practice, the server constructs multiple line segments based on the number of vertices on the graphic outline. For example, if the graphic outline has 30 vertices, more than 29 line segments can be constructed.

[0142] Step 1023b involves filtering multiple line segments to determine the target line segments used to construct the graphic outline of the screen.

[0143] In practice, the server filters multiple line segments constructed from vertices on the graphic outline to determine multiple target line segments for constructing the graphic outline of the screen. For example, the server determines 50 line segments based on 30 vertices of the graphic outline, filters these 50 line segments, and determines 29 line segments as target line segments for constructing the graphic outline of the screen.

[0144] In some embodiments, see Figure 9 , Figure 9 This is a schematic diagram illustrating the method for determining the target line segment provided in an embodiment of this application. Figure 8 Step 1023b shown can also be implemented through steps 401a-403a, in combination with Figure 9 The steps shown are explained.

[0145] Step 401a: The server determines the length of each line segment and constructs a coordinate system using the image as the front view of the image acquisition device to obtain the slope of each line segment in the current coordinate system.

[0146] In practice, the server determines multiple line segments based on the vertices of the graphic contour and calculates the length of each line segment, i.e., determining the distance between two adjacent points as the length of a line segment. Simultaneously, a coordinate system is constructed using the image as the front view of the image acquisition device, and the slope of each line segment is determined within this coordinate system. In practical applications, the length and slope of a line segment can be determined based on the coordinate information of at least two vertices of the line segment in the current coordinate system, where the coordinate system is constructed using the image as the front view of the image acquisition device.

[0147] Step 402a: When the length of a line segment reaches a length threshold, multiple line segments are classified based on the slope to obtain the horizontal and vertical line segments in the corresponding coordinate system.

[0148] In practice, the server determines the length of the line segments and filters them against a preset length threshold, removing those segments whose length does not meet the threshold. The filtered line segments are then used as candidate segments to construct the graphical outline of the screen. The server further groups these candidate line segments based on their slope, obtaining horizontal and vertical line segments corresponding to the coordinate system.

[0149] For example, a coordinate system is constructed using the image as the front view of the image acquisition device, and a straight line with a slope of ±1 in the coordinate system is used as the baseline. Multiple candidate line segments are grouped, and line segments whose straight lines have an inclination angle of [45°, 135°] relative to the horizontal axis of the coordinate system are divided into vertical line segment groups, and line segments whose straight lines have an inclination angle of [0, 45°] and [135°, 180°] relative to the horizontal axis of the coordinate system are divided into horizontal line segment groups.

[0150] Step 403a: Based on the horizontal and vertical line segments, determine the target line segments used to construct the graphic outline of the screen.

[0151] In actual implementation, the server selects target line segments from each horizontal and vertical line segment to construct the graphic outline of the screen.

[0152] In some embodiments, see Figure 10 , Figure 10 This is a schematic diagram illustrating the method for determining the graphic outline of a screen according to an embodiment of this application. Figure 8 Step 1023b shown can also be implemented through steps 401b-403b, in combination with Figure 10 The steps are shown in the diagram.

[0153] Step 401b: The server determines the initial graphic contour of the corresponding screen from at least two graphic contours included in the binarized image.

[0154] In practice, the server determines the initial graphic contour of the screen from at least two graphic contours of the binarized image. For example, see... Figure 5 (4) The image shown includes multiple graphic outlines.

[0155] Step 402b: Construct a coordinate system using the image as the front view of the image acquisition device, and determine the horizontal and vertical center axes of the initial graphic outline in the coordinate system.

[0156] In practice, the server constructs a coordinate system using the image as a front view of the image acquisition device. Based on this coordinate system, the server determines the horizontal and vertical central axes of the initial graphic outline of the screen within that coordinate system.

[0157] For example, see Figure 5 (4) In the figure, the initial graphic outline corresponding to the screen is set to a rectangle, and the server determines the horizontal and vertical center axes of the rectangle relative to the coordinate system.

[0158] Step 403b: Based on the position of each line segment relative to the horizontal central axis, group the line segments to obtain the first horizontal line segment located above the horizontal central axis and the second horizontal line segment located below the horizontal central axis.

[0159] In actual implementation, the server determines the position of each line segment relative to the horizontal central axis, and determines the first horizontal line segment located above the horizontal central axis and the second horizontal line segment located below the horizontal central axis.

[0160] For example, see Figure 5 (4) In the figure, the initial graphic outline corresponding to the screen is set to a rectangle. The server determines the first horizontal line segment located on the upper side of the horizontal central axis, which is to determine the top of the rectangle; and determines the second horizontal line segment located on the lower side of the horizontal central axis, which is to determine the bottom of the rectangle.

[0161] Step 404b: Based on the position of each line segment relative to the longitudinal central axis, group the line segments to obtain the first longitudinal line segment located to the left of the longitudinal central axis and the second longitudinal line segment located to the right of the longitudinal central axis.

[0162] In actual implementation, the server determines the position of each line segment relative to the longitudinal central axis, and determines the first longitudinal line segment located above the longitudinal central axis and the second longitudinal line segment located below the longitudinal central axis.

[0163] For example, see Figure 5 (4) In the figure, the initial graphic outline corresponding to the screen is set to a rectangle. The server determines the first vertical line segment located to the left of the vertical center axis, which is to determine the left side of the rectangle; and determines the second vertical line segment located to the right of the vertical center axis, which is to determine the right side of the rectangle.

[0164] Step 405b: Based on the first horizontal line segment, the second horizontal line segment, the first vertical line segment, and the second vertical line segment, determine the target line segment corresponding to the graphic outline used to construct the screen.

[0165] In actual implementation, the server determines the target line segment corresponding to the graphic outline used to construct the screen based on the determined first horizontal line segment, second horizontal line segment, first vertical line segment, and second vertical line segment.

[0166] In some embodiments, see Figure 11 , Figure 11 This is a schematic diagram illustrating a method for determining the graphic outline of a screen according to an embodiment of this application. Figure 10 Step 405b shown can also be implemented through steps 4051-4054, in combination with Figure 11 The steps are shown in the diagram.

[0167] Step 4051: The server determines the first horizontal midpoint of each first horizontal line segment, and fits each first horizontal line segment based on each first horizontal midpoint to obtain the first target horizontal line segment.

[0168] In practice, to determine the first target horizontal line from multiple first horizontal line segments, the server can fit each first horizontal line segment to its midpoint to obtain the first target horizontal line that forms the outline of the screen graphic. It should be noted that screens are mostly regular quadrilaterals.

[0169] Step 4052: Determine the second horizontal midpoint of each second horizontal line segment, and fit each second horizontal line segment based on each second horizontal midpoint to obtain the second target horizontal line segment.

[0170] In actual implementation, the server can fit each second horizontal line segment based on the horizontal midpoint (second horizontal midpoint) of each second horizontal line segment to obtain the second target horizontal line that constructs the outline of the screen graphic.

[0171] Step 4053: Determine the first longitudinal midpoint of each first longitudinal line segment, and fit each first longitudinal line segment based on each first longitudinal midpoint to obtain the first target longitudinal line segment.

[0172] In actual implementation, the server can fit each first vertical line segment based on the midpoint of each first vertical line segment (first vertical midpoint) to obtain the second target vertical line that constructs the outline of the screen graphic.

[0173] Step 4054: Determine the second vertical midpoint of each second vertical line segment, and fit each second vertical line segment based on each second vertical midpoint to obtain the second target vertical line segment. Then, use the first target horizontal line segment, the second target horizontal line segment, the first target vertical line segment, and the second target vertical line segment as the target line segments corresponding to the graphic contour of the screen.

[0174] In practical implementation, the server can fit each second vertical line segment to its midpoint (second vertical midpoint) to obtain the second target vertical line that constructs the outline of the screen graphic. The graphic outline constructed based on the determined first target horizontal line segment, second target horizontal line segment, first target vertical line segment, and second target vertical line segment is used as the graphic outline of the screen.

[0175] In some embodiments, see Figure 12 , Figure 12 This is a schematic diagram of a screen graphic outline determination method provided in an embodiment of this application. When multiple line segments are filtered and no target line segment for constructing the screen graphic outline is obtained, the server can also... Figure 12 The steps shown determine the graphic outline of the screen in the image.

[0176] Step 401c: The server constructs a coordinate system using the image as the front view of the image acquisition device, and determines the horizontal and vertical center axes of each graphic contour in the binarized image in the coordinate system.

[0177] In actual implementation, when multiple line segments are filtered and no target line segment for constructing the graphic outline of the screen is obtained, the server can construct a coordinate system using the image as the front view of the image acquisition device, and determine the horizontal and vertical central axes of each graphic outline in the binarized graphic relative to the coordinate system.

[0178] Step 402c: Based on each horizontal center axis, determine the first horizontal sub-contour and the second horizontal sub-contour of each graphic contour.

[0179] In actual implementation, the server divides the graphic outline into a first horizontal sub-outline and a second horizontal sub-outline based on the horizontal central axis.

[0180] Step 403c: Based on each longitudinal center axis, determine the first longitudinal sub-contour and the second longitudinal sub-contour of each graphic contour.

[0181] In actual implementation, the server divides the graphic outline into a first vertical sub-outline and a second vertical sub-outline based on the vertical central axis.

[0182] Step 404c: Determine the graphic outline of the screen in the image based on the first horizontal sub-outline, the second horizontal sub-outline, the first vertical sub-outline, and the second vertical sub-outline.

[0183] In practice, for the first horizontal sub-contour, the server acquires multiple line segments comprising the first horizontal sub-contour, determines the slope of each line segment, calculates the average slope of each line segment, and constructs a smooth straight line corresponding to the first horizontal sub-contour based on this average slope (the slope of this smooth straight line is the aforementioned average slope). Similarly, the server can determine smooth straight lines corresponding to the second horizontal sub-contour, the first vertical sub-contour, and the second vertical sub-contour. Finally, based on the smooth straight lines corresponding to each sub-contour, the server determines the graphic contour of the screen in the image.

[0184] Step 1024a: Determine the graphic outline of the screen in the image based on the target line segment.

[0185] In practice, the server will use the graphic outline constructed based on the target line segment as the graphic outline of the screen in the image.

[0186] In step 103, the screen image is corrected to obtain a front view image when the image acquisition device is looking directly at the screen to acquire the image.

[0187] In some embodiments, the server can perform image correction on the screen image in the following manner: the server obtains a first coordinate system corresponding to the image acquisition device when the image is used as the front view of the image acquisition device, and a second coordinate system corresponding to the screen when the image acquisition device is looking at the screen to acquire the image; based on the first coordinate system and the second coordinate system, the server performs coordinate transformation on each pixel in the screen image to obtain the front view image when the corresponding image acquisition device is looking at the screen to acquire the image.

[0188] In practice, the server obtains the coordinate information of each pixel in the screen image in the first coordinate system and converts it into coordinate information in the second coordinate system. Based on the converted pixels, the front view of the image is determined.

[0189] In step 104, the position of the light spot in the front view image is determined, and based on the position, an interactive response is executed for the interactive operation.

[0190] In some embodiments, when the image includes at least two light spots, the server can also determine the position of the light spots in the frontal view image by: the server obtaining the value of the light spot parameter of each light spot, wherein the light spot parameter includes at least one of the light spot shape and the light spot brightness; based on the light spot parameter, the light spot with the largest value of the light spot parameter is selected as the target light spot, and the position of the target light spot in the frontal view image is determined.

[0191] In practice, the server can compare the similarity between the shape of each light spot and the shape of a preset target light spot, and select the spot with the highest similarity as the target light spot. The server can also compare the brightness of each light spot and select the brightest (highest) spot as the target light spot. In other words, the server can eliminate interfering light spots in the image based on at least one of the light spot shape and brightness.

[0192] In some embodiments, when the number of images is multiple frames, the server can also determine the position of the light spot in the front view image by: the server obtaining the position information of the light spot in each image; determining the running trajectory of the light spot based on the position information; taking the light spot on the running trajectory as the target light spot, and determining the position of the target light spot in the front view image.

[0193] In practice, the server can determine the position of each spot in multiple consecutive frames of images and, based on the position, determine the trajectory of each spot. If a position point is not on the trajectory, the spot corresponding to that position point can be excluded, thus identifying that spot as an interfering spot.

[0194] In some embodiments, the server can execute an interactive response to an interactive operation based on the location of the light spot: when the interactive operation is a graphics drawing operation, a dotted drawing of the corresponding light spot is performed at the location; when the interactive operation is a content triggering operation, the triggering result of the content corresponding to the triggering operation is output.

[0195] In actual implementation, after the server determines the position of the light spot in the front view, it can execute an interactive response for the interactive operation at the current position.

[0196] For example, Figures 13A-13B This is a schematic diagram of the light spot-based interactive operation provided in an embodiment of this application. See also... Figure 13A In the interactive operation, which is based on drawing with a laser spot (which can be understood as drawing with a laser pointer), the server, after determining the position of the laser spot, can control the laser spot to perform drawing operations on the screen. Figure 13A (As shown in number 1) After the laser spot is projected onto the screen and its position is obtained, it can be drawn based on its trajectory on the screen. See also Figure 13B Interactive operations can also simulate hitting a target during shooting. After the server determines the location of the laser spot, it can interact with the content at that location (which can be understood as clicking to select the content at that location). Figure 13B The middle screen displays a simple, simulated shooting game application where players can use a handheld laser device (such as a laser pointer) to hit the game targets shown in the video. Figure 13B (The laser spot shown in number 1 hits the game content).

[0197] By applying the embodiments of this application, the graphic outline of the screen in the image including the screen and the light spot acquired by the image acquisition device is determined, the corresponding screen image is determined, and the screen image is corrected to obtain a front view image when the image acquisition device is looking directly at the screen to acquire the image. The target position of the light spot is determined in the front view image. In this way, the position of the light spot can be accurately located in the screen image from various angles, improving the accuracy of the light spot positioning. Finally, based on the position of the light spot, an interactive response for the interactive operation is executed, thus increasing the fun of the interactive operation based on the light spot.

[0198] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0199] In related technologies, infrared cameras can be used to enable users to interact with content displayed on a screen. For example, in gaming applications, this allows players to interact with in-game content via the screen to complete interactive gameplay. See also... Figure 14 , Figure 14 This is a schematic diagram of game application interaction provided by related technologies, which can be implemented through the following steps: 1. Use an infrared camera to capture the player's image; 2. The server processes the captured infrared image to obtain the coordinate information of the player's skeleton; 3. Use the player's specified body part (hips or hands) as input coordinate information; 4. Based on the player's coordinate information, apply the coordinate information to the game application displayed on the screen, so as to control the left and right movement, jumping, and other actions of the corresponding objects on the screen based on the player's left and right movements, jumping, etc. Capturing the player with an infrared camera can obtain more information, such as the position of the hand and foot bones. The main application scenarios can include two types: one is to obtain the player's position in the current infrared image and determine the player's left and right movement, simple actions such as jumping and crouching, which accounts for the majority of application scenarios; the other is to determine the current action based on the position of various parts of the player's skeleton. However, due to the impact of obstruction and dark clothing on accuracy, only a general judgment is usually made. In practical applications, the above method has the following problems: 1. The position information is not precise enough, for example, it is difficult for players to control their bodies to draw a circle; 2. The position information is not stable enough and is prone to drifting, because players may be obstructed or turn around during movement; 3. Only horizontal movement is possible, and only simple actions such as jumping and crouching can be made in the vertical direction, because the body cannot move completely freely due to the influence of gravity.

[0200] Based on this, embodiments of this application provide a method for locating a laser spot. This method enables interaction with content of a target application on a screen using a laser pointer and a regular camera. In other words, it allows the laser pointer to write on the screen like a mouse, adding a fun offline interaction method and can also be applied to offline marketing scenarios. Through the laser spot location method provided in this application, the accuracy of player input location information can be significantly improved, even allowing for drawing and writing; the location information is much more stable, even simulating free shooting; and location information can be input arbitrarily in both horizontal and vertical directions.

[0201] In practical applications, when implementing the spot positioning method provided in the embodiments of this application, the user needs to hold a light emitting device (such as a laser pointer).

[0202] Next, we will explain from the product side, see [link / reference]. Figure 15 , Figure 15 This is a schematic diagram of the interactive operation based on the light emitting device provided in the embodiment of this application. In the figure, (1) the player holds a laser pointer and rotates the camera to take pictures of the screen on the side of the screen or in front of the player; (2) the video stream (i.e., multiple consecutive images) taken by the camera on the screen is obtained; the position coordinates of the laser point in the captured screen image are obtained, and the position in the image is calculated to correspond to the coordinates in the application or game; (3) the laser pointer spot coordinates are obtained, and the spot coordinates are applied to the target application or target game.

[0203] Next, we will explain from a technical perspective. (See...) Figure 16A , Figure 16A This is a schematic diagram illustrating the complete process of spot positioning provided in the embodiments of this application, combined with... Figure 16A The steps shown are explained.

[0204] Step 501: The camera captures at least one frame of image of the screen, the image including the laser spot and the screen.

[0205] In practice, the user holds a laser pointer and rotates the camera to capture an image of the screen from the side or in front of the player. Because the camera's position is arbitrary, the shape of the screen in the captured image will be distorted from various angles. For example, see... Figure 16B , Figure 16B This is a visual schematic diagram of the spot positioning method provided in the embodiments of this application. Figure 16B (1) shows a user holding a laser pointer, standing in front of the projection screen, and the laser pointer projects laser light onto the screen, which appears as a laser spot (also called a laser dot).

[0206] Step 502: The server receives at least one image frame uploaded by the camera device.

[0207] In practice, the server can be the backend server for the target application or a dedicated server for image recognition. See also Figure 16B , Figure 16B (2) shows the screen image captured by the camera.

[0208] Step 503: The server preprocesses at least one frame of the image to obtain a smoothed image.

[0209] In practice, image preprocessing operations can include image smoothing and noise reduction (i.e., ignoring details through blurring), denoising using mean filtering, and reducing the level of detail. For example, see... Figure 16B , Figure 16B (3) shows the image after smoothing.

[0210] Step 504: Binarize the smoothed image to obtain a binarized image.

[0211] In practical implementation, binarized images are used to highlight the outlines of various physical objects in the captured screen image. Since the projection brightness is higher than the ambient brightness in real-world applications, interference from lights, luminous objects, and other light sources is permissible. For example, see... Figure 16B , Figure 16B (4) The image shown is a binarized image obtained after binarization.

[0212] Step 505: Determine the multiple connected color patches included in the binarized image, and the contour points corresponding to the contours of each connected color patch.

[0213] In practical implementation, connected color patches and their outlines can be determined as follows: The server scans the entire image to determine each connected color patch and the outline points corresponding to the outlines of each connected color patch; the outline with the largest area among the outlines of each connected color patch is obtained; and the corresponding convex polygon is determined using the outline points of the largest area outline. In practical applications, the outline corresponding to the screen is usually a convex polygon. For example, see... Figure 16B , Figure 16B (4) shows multiple connected color blocks in the image (each white area in the image).

[0214] Step 506: Determine at least one straight line in the binarized image based on the contour points of each connected color block.

[0215] In practical implementation, all straight lines in the binarized image can be determined as follows: Calculate the length of the line segment formed by all adjacent points in the convex hull and the slope of the line to which the line segment belongs; remove adjacent points whose distance is less than a distance threshold (i.e., remove adjacent points that are too close; the corners of the quadrilateral screen outline will be rounded after noise reduction); determine the slope of each straight line based on the coordinate system used by the camera, and group the straight lines in the binarized image based on each slope into horizontal straight lines relative to the coordinate system used by the camera and vertical straight lines relative to the coordinate system used by the camera, and extract the average slope of the horizontal straight lines and the average slope of the vertical straight lines respectively. For example, see [link to example]. Figure 16B , Figure 16B (5) The graphic outlines of each object in the image shown. Figure 16B (6) shows a schematic diagram of multiple straight lines determined by the contour points of the graphic and the straight lines being grouped.

[0216] Step 507: Filter each horizontal line and each vertical line to extract the quadrilateral corresponding to the screen in the binarized image.

[0217] In practical implementation, the server can determine the quadrilaterals of the screen in the binarized image by: determining the top horizontal line, bottom horizontal line, left vertical line, and right vertical line of the quadrilateral corresponding to the screen based on the position of line segments and horizontal and vertical grouping. The entire process can be understood as recursively generating four straight lines by continuously taking the midpoints of line segments. In practical applications, the lines can be filtered using preset filtering rules, such as: no lines on the screen, lines that are considered to form a quadrilateral by adjacent lines, lines that are considered to have the largest possible area, and lines that are not on the screen edge.

[0218] In actual implementation, if there are no results or the results are obviously abnormal (the area is too small, the brightness inside the quadrilateral is too low), then extract the color block with the largest area, extract the protruding part of the outline, and extract four straight lines based on the average slope of each straight line corresponding to the outline graphic.

[0219] Step 508: Perform perspective transformation on the quadrilateral corresponding to the screen in the image to obtain the front view image when the corresponding camera is looking at the screen and acquiring the image.

[0220] In practice, since the screen in the image may be deformed at various angles, after obtaining the quadrilateral corresponding to the screen in the image, the coordinates of each pixel in the quadrilateral area can be transformed to obtain the front view image of the screen in the image.

[0221] Step 509: Eliminate interfering light spots in the front view image to determine the laser spot.

[0222] In practice, the laser beam hits the projection screen with the highest brightness, allowing for the presence of other interference points. Therefore, interference points in the screen image can be eliminated using the following methods: 1) Compare the hue of the laser point with that of the interference points to eliminate interference points. Generally, the hue of the laser point is considered the brightest in the binary image of the screen. 2) Eliminate interference points based on the laser's trajectory in the previous frame. For example, if the point is in the upper right corner in the current frame but in the lower left corner in the previous frame, it indicates that the point does not conform to the normal trajectory and is considered an interference point. 3) Obtain the shape of each spot in the binary image and compare each shape with the preset target shape of the laser point. The spot with the most similar shape to the target shape is taken as the laser point.

[0223] Step 510: After eliminating interference points, directly obtain the position of the laser spot on the front view screen.

[0224] After completing the above steps 501-510, the correct position of the laser spot on the screen can be determined.

[0225] In actual implementation, see Figure 17 , Figure 17 This is a schematic diagram of the spot positioning method provided in the embodiments of this application. Steps 501-510 above can be achieved through... Figure 17 In summary: Step 1, acquire an image from the camera; Step 2, denoise and reduce detail; Step 3, binarize the image; Step 4, extract edges of objects in the binarized image; Step 5, detect straight lines in the image; Step 6, filter the straight lines and extract quadrilaterals; Step 7, determine if quadrilaterals can be extracted normally; if not, repeat Step 1; if extraction is successful, proceed to Step 8, determine if the quadrilateral's brightness is higher than the average; if so, proceed to Step 9, determine if the quadrilateral's area reaches 50%; if so, proceed to Step 10, obtain the laser screen coordinates; Step 11, perform perspective transformation on the screen coordinates; Step 12, detect the color blocks of the laser spot; Step 13, determine if the color blocks are close to a circle; if so, proceed to Step 14, determine if the laser spot's position deviates from its trajectory; otherwise, proceed to Step 15, generate player operation events based on the spot's position; Step 16, send the laser spot's position to the target application.

[0226] By applying the embodiments of this application, a laser pointer can be used to write on a projected screen like a mouse, adding a fun offline interactive method that can be applied to offline marketing scenarios; the accuracy of the player's input location information can be greatly improved, and even drawing and writing can be performed; the location information is relatively stable, and even free shooting can be simulated; and the location information can be freely input in both horizontal and vertical directions.

[0227] In this application embodiment, data such as user information is involved. When this application embodiment is applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0228] The following description continues to illustrate the exemplary structure of the laser spot positioning device 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules in the processing device 555 storing services in the virtual scene in the memory 550 may include:

[0229] The acquisition module 5551 is used to acquire at least one frame of image obtained by the image acquisition device from the screen during the content display process; wherein, the image includes: a light spot generated by the light emitting device performing an interactive operation on the content displayed on the screen by emitting light;

[0230] The determining module 5552 is used to determine the graphic outline of the screen in the image, and to obtain the screen image corresponding to the screen based on the graphic outline of the screen.

[0231] The correction module 5553 is used to perform image correction on the screen image to obtain a front view image corresponding to the image acquisition device when it is looking at the screen and acquiring the image.

[0232] The positioning module 5554 is used to determine the position of the light spot in the front view image, and based on the position, execute an interactive response for the interactive operation.

[0233] In some embodiments, the determining module is further configured to perform image recognition on the image to obtain the region contours corresponding to each object entity included in the image; obtain the region area of ​​the region contour corresponding to each object entity, and determine the region contour with the largest region area as the target region contour, and the corresponding object entity as the screen; obtain the vertices on the target region contour, and determine the graphic contour of the screen in the image based on the vertices.

[0234] In some embodiments, the determining module is further configured to perform binarization processing on the image to obtain a binarized image; perform connectivity analysis on the binarized image to obtain the image regions corresponding to each object entity included in the image; extract the contour points of the edges of each image region, and determine the region contours corresponding to each object entity included in the image based on the contour points of the edges of each image region.

[0235] In some embodiments, the determining module is further configured to determine the distance between adjacent vertices on the target region contour; based on the distance between adjacent vertices, filter the multiple vertices included in the target region contour to obtain multiple target vertices; connect the multiple target vertices to obtain the initial contour of the screen, and smooth the initial contour to obtain the graphic contour of the screen in the image.

[0236] In some embodiments, the determining module is further configured to perform binarization processing on the image to obtain a binarized image including at least two graphic contours, the graphic contours including multiple line segments; filter the multiple line segments to determine target line segments for constructing the graphic contours of the screen, and determine the graphic contours of the screen in the image based on the target line segments.

[0237] In some embodiments, the determining module is further configured to determine the length of each line segment, construct a coordinate system using the image as the front view of the image acquisition device, and obtain the slope of each line segment in the coordinate system; when the length of the line segment reaches a length threshold, classify multiple line segments based on the slope to obtain horizontal line segments and vertical line segments corresponding to the coordinate system; and determine the target line segments for constructing the graphic outline of the screen based on the horizontal and vertical line segments.

[0238] In some embodiments, the determining module is further configured to: determine an initial graphic contour corresponding to the screen from at least two graphic contours included in the binarized image; construct a coordinate system using the image as a front view of the image acquisition device, and determine the horizontal and vertical central axes of the initial graphic contour in the coordinate system; group the line segments based on their positions relative to the horizontal central axis to obtain a first horizontal line segment located above the horizontal central axis and a second horizontal line segment located below the horizontal central axis; group the line segments based on their positions relative to the vertical central axis to obtain a first vertical line segment located to the left of the vertical central axis and a second vertical line segment located to the right of the vertical central axis; and determine a target line segment corresponding to the graphic contour used to construct the screen based on the first horizontal line segment, the second horizontal line segment, the first vertical line segment, and the second vertical line segment.

[0239] In some embodiments, the determining module is further configured to, when filtering multiple line segments and no target line segment for constructing the graphic contour of the screen is obtained, construct a coordinate system using the image as the front view of the image acquisition device, and determine the horizontal and vertical central axes of each graphic contour in the binary image in the coordinate system; determine a first horizontal sub-contour and a second horizontal sub-contour of each graphic contour based on each horizontal central axis; determine a first vertical sub-contour and a second vertical sub-contour of each graphic contour based on each vertical central axis; and determine the graphic contour of the screen in the image based on the first horizontal sub-contour, the second horizontal sub-contour, the first vertical sub-contour, and the second vertical sub-contour.

[0240] In some embodiments, the correction module is further configured to obtain a first coordinate system corresponding to the image acquisition device when the image is used as the front view of the image acquisition device, and a second coordinate system corresponding to the screen when the image acquisition device is looking at the screen and performing image acquisition; based on the first coordinate system and the second coordinate system, perform coordinate transformation on each pixel in the screen image to obtain a front view image corresponding to the image acquisition device looking at the screen and performing image acquisition.

[0241] In some embodiments, when the image includes at least two light spots, the positioning module is further configured to obtain the values ​​of light spot parameters for each light spot, the light spot parameters including at least one of light spot shape and light spot brightness; based on the light spot parameters, the light spot with the largest value of the light spot parameter is selected as the target light spot, and the position of the target light spot in the front view image is determined.

[0242] In some embodiments, when the number of images is multiple frames, the positioning module is further configured to acquire the position information of the light spot in each of the images; determine the running trajectory of the light spot based on the position information; take the light spot on the running trajectory as the target light spot, and determine the position of the target light spot in the front view image.

[0243] In some embodiments, the positioning module is further configured to, when the interaction operation is a graphics drawing operation, draw a dotted pattern corresponding to the light spot at the location; and when the interaction operation is a trigger operation for the content, output the trigger result of the trigger operation corresponding to the content.

[0244] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the spot positioning method described in this application.

[0245] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the spot localization method provided in this application. For example, ... Figure 3 The method for locating the light spot is shown.

[0246] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0247] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0248] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0249] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0250] In summary, the embodiments of this application can improve the accuracy of spot positioning and increase the fun of interactive operations performed based on the spot.

[0251] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method of locating a light spot, characterized in that The method comprises: During content display on a screen, at least one frame of image obtained by an image acquisition device performing image acquisition on the screen is acquired; The image comprises a light spot generated by a light emitting device performing an interactive operation on the content displayed on the screen by emitting light; The image is subjected to binaryzation processing to obtain a binaryzation image comprising at least two graphic contours, and the graphic contours comprise a plurality of line segments; From the at least two graphic contours of the binaryzation image, an initial graphic contour corresponding to the screen is determined; A coordinate system is constructed with the image as an orthographic view of the image acquisition device, and a horizontal center axis and a vertical center axis of the initial graphic contour in the coordinate system are determined; Based on the positions of the line segments relative to the horizontal center axis and the vertical center axis, the line segments are grouped to obtain a first horizontal line segment located on the upper side of the horizontal center axis, a second horizontal line segment located on the lower side of the horizontal center axis, a first vertical line segment located on the left side of the vertical center axis, and a second vertical line segment located on the right side of the vertical center axis; Midpoints corresponding to the first horizontal line segment, the second horizontal line segment, the first vertical line segment, and the second vertical line segment are determined respectively, and the first horizontal line segment, the second horizontal line segment, the first vertical line segment, and the second vertical line segment are fitted based on the midpoints respectively to obtain a first target horizontal line segment, a second target horizontal line segment, a first target vertical line segment, and a second target vertical line segment as target line segments corresponding to the graphic contour of the screen; The graphic contour of the screen is determined based on the target line segments; When the fitting based on the midpoints does not obtain the first target horizontal line segment, the second target horizontal line segment, the first target vertical line segment, and the second target vertical line segment, horizontal center axes and vertical center axes of each graphic contour in the coordinate system are determined in the binaryzation image; first horizontal sub-contours and second horizontal sub-contours of each graphic contour are determined based on each horizontal center axis; first vertical sub-contours and second vertical sub-contours of each graphic contour are determined based on each vertical center axis; a smooth straight line corresponding to each is constructed based on the average slope of the plurality of line segments included in the first horizontal sub-contour, the second horizontal sub-contour, the first vertical sub-contour, and the second vertical sub-contour; and the graphic contour of the screen is determined based on the smooth straight line; Based on the graphic contour of the screen, a screen image corresponding to the screen is acquired; The screen image is subjected to image correction to obtain an orthographic view image corresponding to the image acquisition device performing image acquisition on the screen in an orthographic view; The position of the light spot in the orthographic view image is determined, and an interactive response to the interactive operation is performed based on the position.

2. The method of claim 1, wherein, The method further comprises: The image is subjected to image recognition to obtain area contours corresponding to each object entity included in the image; The area of each area contour corresponding to each object entity is acquired, and the area contour with the largest area is determined as a target area contour, and the corresponding object entity is determined as the screen. acquire a vertex on the target region contour, and determine a graphic contour of the screen in the image based on the vertex.

3. The method of claim 2, wherein, The image recognition on the image to obtain region contours corresponding to each object entity included in the image comprises: performing binaryzation processing on the image to obtain a binaryzation image; performing connectivity analysis on the binaryzation image to obtain image regions corresponding to each object entity included in the image; extracting contour points of edges of each image region, and determining region contours corresponding to each object entity included in the image based on the contour points of edges of each image region.

4. The method of claim 2, wherein, The determination of the graphic contour of the screen in the image based on the vertex comprises: determining distances between adjacent vertices on the target region contour; performing screening on a plurality of vertices included in the target region contour based on the distances between the adjacent vertices to obtain a plurality of target vertices; connecting the plurality of target vertices to obtain an initial contour of the screen, and performing smoothing processing on the initial contour to obtain the graphic contour of the screen in the image.

5. The method of claim 1, wherein, The method further comprises: determining lengths of each line segment, and constructing a coordinate system with the image as an orthographic view of the image acquisition device to acquire slopes of each line segment in the coordinate system; when the length of the line segment reaches a length threshold, classifying a plurality of the line segments based on the slopes to obtain horizontal line segments and vertical line segments corresponding to the coordinate system; determining target line segments for constructing the graphic contour of the screen based on the horizontal line segments and the vertical line segments.

6. The method of claim 1, wherein, The image correction on the screen image to obtain an orthographic view corresponding to image acquisition by the image acquisition device viewing the screen orthographically comprises: acquiring a first coordinate system corresponding to the image acquisition device when the image is taken as an orthographic view of the image acquisition device, and a second coordinate system corresponding to the screen when the image acquisition device views the screen for image acquisition; performing coordinate conversion on each pixel point in the screen image based on the first coordinate system and the second coordinate system to obtain an orthographic view corresponding to image acquisition by the image acquisition device viewing the screen orthographically.

7. The method of claim 1, wherein, When the image includes at least two light spots, the determination of the position of the light spot in the orthographic view comprises: acquiring values of light spot parameters of each light spot, the light spot parameters including at least one of light spot shape and light spot brightness; based on the light spot parameters, screening a light spot with the largest value of the light spot parameters as a target light spot, and determining the position of the target light spot in the orthographic view.

8. The method of claim 1, wherein, When the number of images is multiple frames, the determination of the position of the light spot in the orthographic view comprises: acquiring position information of the light spot in each image; based on each position information, determining a running track of the light spot; screening a light spot on the running track as a target light spot, and determining the position of the target light spot in the orthographic view.

9. The method of claim 1, wherein, The execution of the interactive response to the interactive operation based on the position comprises: when the interactive operation is a graphical drawing operation, performing point drawing corresponding to the light spot at the position; when the interactive operation is a triggering operation on the content, outputting a triggering result of the triggering operation on the content.

10. A device for positioning a light spot, characterized in that The device comprises: an acquisition module, configured to acquire at least one frame of image obtained by an image acquisition device performing image acquisition on a screen during content display on the screen; wherein the image comprises a light spot generated by a light emitting device performing an interactive operation on content displayed on the screen by emitting light; a determination module, configured to perform binaryzation processing on the image to obtain a binaryzation image comprising at least two graphical contours, the graphical contours comprising a plurality of line segments; determine an initial graphical contour corresponding to the screen from the at least two graphical contours of the binaryzation image; construct a coordinate system with the image as an orthographic view of the image acquisition device and determine a horizontal central axis and a vertical central axis of the initial graphical contour in the coordinate system; group the line segments based on their positions relative to the horizontal central axis and the vertical central axis to obtain a first horizontal line segment located on the upper side of the horizontal central axis, a second horizontal line segment located on the lower side of the horizontal central axis, a first vertical line segment located on the left side of the vertical central axis, and a second vertical line segment located on the right side of the vertical central axis; determine the midpoints corresponding to the first horizontal line segment, the second horizontal line segment, the first vertical line segment, and the second vertical line segment respectively, and fit the first horizontal line segment, the second horizontal line segment, the first vertical line segment, and the second vertical line segment based on the midpoints respectively to obtain a first target horizontal line segment, a second target horizontal line segment, a first target vertical line segment, and a second target vertical line segment as target line segments corresponding to the graphical contour of the screen; determine the graphical contour of the screen based on the target line segments; when the fitting based on the midpoints does not obtain the first target horizontal line segment, the second target horizontal line segment, the first target vertical line segment, and the second target vertical line segment, determine the horizontal central axes and the vertical central axes of the graphical contours in the coordinate system; determine a first horizontal sub-contour and a second horizontal sub-contour of each graphical contour based on the horizontal central axes; determine a first vertical sub-contour and a second vertical sub-contour of each graphical contour based on the vertical central axes; construct a smooth straight line corresponding to the graphical contour based on the average slope of the plurality of line segments included in the first horizontal sub-contour, the second horizontal sub-contour, the first vertical sub-contour, and the second vertical sub-contour; determine the graphical contour of the screen based on the smooth straight line; and acquire a screen image corresponding to the screen based on the graphical contour of the screen. a correction module, configured to perform image correction on the screen image to obtain an orthographic view image corresponding to the image acquisition device performing image acquisition on the screen in an orthographic view. The positioning module is configured to determine a position of the light spot in the front view image, and perform an interaction response for the interaction operation based on the position.

11. The apparatus of claim 10, wherein, The correction module is further configured to obtain a first coordinate system corresponding to the image acquisition device when the image acquisition device is in a front view state, and a second coordinate system corresponding to the screen when the image acquisition device is in a front view state of the screen; and perform coordinate conversion on each pixel point in the screen image based on the first coordinate system and the second coordinate system, to obtain a front view image corresponding to the image acquisition device in the front view state of the screen.

12. The apparatus of claim 10, wherein, When the image includes at least two light spots, The positioning module is further configured to obtain a value of a light spot parameter of each light spot, the light spot parameter including at least one of a light spot shape and a light spot brightness; select a light spot with a maximum value of the light spot parameter as a target light spot based on the light spot parameter; and determine a position of the target light spot in the front view image.

13. An electronic device, comprising: The electronic device includes: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the light spot positioning method of any one of claims 1 to 11.

14. A computer-readable storage medium storing executable instructions, the method comprising: The executable instructions, when executed by the processor, implement the light spot positioning method of any one of claims 1 to 9.

15. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions, when executed by the processor, implement the light spot positioning method of any one of claims 1 to 9.

Images