Smart device for touch screen aware control

Abstract

The application relates to the touch control technical field, and particularly provides a kind of intelligent device for touch screen sensing control, to solve how to conveniently and reliably sense and touch the touch screen.The intelligent device of the application includes film-like matrix and photosensitive unit array, touch unit array and control circuit arranged on the film-like matrix;The film-like matrix is used to attach to the surface of the touch screen;The photosensitive unit array is used to collect the display light signal emitted by the touch screen;The touch unit array is used to output touch analog signal to the touch screen;The control circuit is configured to generate the virtual screen of the touch screen according to the display light signal, identify the screen content of the virtual screen, generate touch action according to the screen content and the preset task script;According to the touch action, the touch unit array outputs the touch analog signal to the touch screen to execute the touch action and verify the execution result.Based on the above structure, the intelligent device can realize sensing and touching the touch screen simultaneously.

Description

Technical Field

[0001] This application relates to the field of touch technology, specifically providing a smart device for touch screen sensing and control. Background Technology

[0002] In scenarios such as testing, operation and maintenance, and management of electronic devices, it may be necessary to perform long-term, repetitive, and procedural automated operations on the touch screen of electronic devices. Currently, the main automated operation methods for touch screens include the following: (1) Automating the operation of the touch screen based on the software interface of the electronic device. If the electronic device (such as an electronic device with a closed operating system) does not open its software interface, this method cannot be used. (2) Collecting the screen content of the touch screen through a camera and controlling the robotic arm and other mechanisms to perform touch operations on the touch screen based on the screen content. This method is complex to deploy and has high maintenance costs. The camera is easily affected by factors such as lighting when collecting screen content. (3) Integrating photoelectric and touch sensors inside the touch screen and realizing screen content perception and control through these built-in sensors. The built-in sensors are part of the touch screen's own structure and cannot be disassembled. They cannot be used as external accessories for automated operation of the touch screen.

[0003] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention

[0004] This application aims to solve the above-mentioned technical problems, namely, to solve or at least partially solve the following technical problems: how to conveniently and reliably sense and control a touch screen in order to achieve automated operation of the touch screen.

[0005] Specifically, this application provides an intelligent device for touch screen sensing control. The intelligent device includes a film substrate and a photosensitive unit array, a touch unit array, and a control circuit disposed on the film substrate. The control circuit is connected to the photosensitive unit array and the touch unit array, respectively.

[0006] The film-like substrate is used to be attached to the surface of the touch screen;

[0007] The photosensitive unit array is used to collect the display light signal emitted by the touch screen;

[0008] The touch unit array is used to output touch analog signals to the touch screen;

[0009] The control circuit is configured to perform the following operations:

[0010] A virtual screen of the touch screen is generated based on the display light signal, the screen content of the virtual screen is identified, and touch actions are generated based on the screen content and a preset task script. The preset task script is used to indicate the automated operation process of the touch screen.

[0011] The touch unit array is controlled to output the touch simulation signal to the touch screen according to the touch action to execute the touch action, and the screen content of the virtual screen is identified after the touch action is executed, and the execution result of the touch action is verified according to the screen content.

[0012] In one technical solution of the above-mentioned smart device, the film-like substrate includes a screen coverage area and an edge functional area. The screen coverage area is used to cover the display area of ​​the touch screen, and the edge functional area is disposed at the edge of the screen coverage area.

[0013] The photosensitive unit array and the touch unit array are disposed in the screen coverage area;

[0014] The control circuit is located in the edge functional area.

[0015] In one technical solution of the aforementioned smart device, the photosensitive unit array and the touch unit array are disposed in the screen coverage area, including:

[0016] The photosensitive unit array and the touch unit array are arranged in layers in the screen coverage area.

[0017] In one technical solution of the aforementioned smart device, the photosensitive unit array and the touch unit array are disposed in the screen coverage area, including:

[0018] The photosensitive units in the photosensitive unit array and the touch units in the touch unit array are arranged sequentially at intervals on the same plane within the screen coverage area.

[0019] In one technical solution of the aforementioned smart device, the photosensitive unit array and the touch unit array are disposed in the screen coverage area, including:

[0020] The screen coverage area is divided into multiple partitions, and multiple photosensitive units in the photosensitive unit array and multiple touch units in the touch unit array are respectively set in each partition.

[0021] In one technical solution of the aforementioned smart device, generating the virtual screen of the touchscreen based on the display light signal includes:

[0022] An image is generated based on the displayed light signal, and the virtual screen is determined based on the image.

[0023] In one technical solution of the aforementioned smart device, the step of generating touch actions based on the screen content and a preset task script includes:

[0024] The automated operation process indicated by the preset task script includes multiple operation steps, execution conditions for each operation step, and execution actions. If the screen content is found to meet the execution conditions of an operation step, the touch action is generated according to the execution action of the operation step.

[0025] In one technical solution of the aforementioned smart device, verifying the execution result of the touch action based on the screen content includes:

[0026] If the touch action fails, then one of the following will be executed:

[0027] Adjust the touch action, and based on the adjusted touch action, re-control the touch unit array to output touch analog signals to the touch screen;

[0028] The screen content of the virtual screen is re-identified, and a touch action is generated again based on the re-identified screen content and the preset task script. The touch unit array is then controlled to output a touch simulation signal to the touch screen based on the regenerated touch action.

[0029] In one technical solution of the aforementioned smart device, the control circuit is further configured to perform the following operations:

[0030] Within the photosensitive sampling time window, the photosensitive unit array is controlled to collect the display light signal emitted by the touch screen, and the touch unit array is shielded.

[0031] Within the touch execution time window, the touch unit array is controlled to output touch simulation signals to the touch screen, and the photosensitive unit array is shielded.

[0032] In one technical solution of the aforementioned smart device, the control circuit is further configured to perform the following operations:

[0033] Send the display light signal collected by the photosensitive unit array to the server;

[0034] Receive touch commands sent by the server;

[0035] The touch command is parsed to obtain the touch action, and the touch unit array is controlled to output touch analog signals to the touch screen according to the touch action to execute the touch action. After the touch action is executed, execution completion information is sent to the server.

[0036] The touch action is generated by the server based on the screen content of the virtual screen and a preset task script. The virtual screen is generated by the server based on the display light signal. The execution completion information is used to instruct the server to identify the screen content of the virtual screen and verify the execution result of the touch action based on the screen content.

[0037] The above-described technical solutions of this application have at least one or more of the following beneficial effects:

[0038] In one technical solution of the intelligent device for touch screen sensing control provided in this application, the intelligent device may include a film substrate and a photosensitive unit array, a touch unit array, and a control circuit disposed on the film substrate. The control circuit is connected to the photosensitive unit array and the touch unit array respectively. The film substrate is used to be attached to the surface of the touch screen. The photosensitive unit array is used to collect the display light signal emitted by the touch screen. The touch unit array is used to output touch analog signals to the touch screen. The control circuit is configured to perform the following operations: generate a virtual screen of the touch screen based on the display light signal, identify the screen content of the virtual screen, generate touch actions based on the screen content and a preset task script, the preset task script being used to instruct the automated operation process of the touch screen; control the touch unit array to output touch analog signals to the touch screen to execute the touch action, and identify the screen content of the virtual screen after executing the touch action, and verify the execution result of the touch action based on the screen content.

[0039] In the above implementation scheme, the screen content of the virtual screen can describe the screen content displayed on the touch screen. Therefore, acquiring the screen content of the virtual screen through the photosensitive unit array and control circuit is equivalent to realizing the perception of the screen content of the touch screen. In addition, the touch unit array outputs touch analog signals to the touch screen, realizing the execution of touch actions on the touch screen. It can be seen that the above implementation scheme simultaneously realizes the perception and touch control of the touch screen. Compared with the prior art, the above implementation scheme does not need to call the software interface of the electronic device (i.e., the electronic device equipped with the touch screen). Regardless of whether the electronic device has access to the software interface, it can perceive and control the touch screen. The above implementation scheme also does not use mechanisms such as robotic arms to control the touch screen, which can reduce the complexity of the scheme and maintenance costs. The above implementation scheme also does not need to integrate photoelectric and touch sensors inside the touch screen, overcoming the defect that it cannot be used as an external accessory for automated operation of the touch screen. Attached Figure Description

[0040] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:

[0041] Figure 1 This is a schematic diagram of the main structure of a smart device used for touch screen sensing control in some embodiments of this application;

[0042] Figure 2 This is a schematic diagram of the partitioning of the membrane substrate in some embodiments of this application. Detailed Implementation

[0043] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0044] In some embodiments of this application, the smart device for touch screen sensing control may include a film substrate and a photosensitive unit array, a touch unit array, and a control circuit disposed on the film substrate. The control circuit is connected to the photosensitive unit array and the touch unit array, respectively. The film substrate, the photosensitive unit array, the touch unit array, and the control circuit are described below.

[0045] 1. Description of the membrane matrix.

[0046] In this embodiment, the film-like substrate is used to be attached to the surface of the touch screen, and after being attached to the surface of the touch screen, the film-like substrate can cover the display area of ​​the touch screen.

[0047] The embodiments of this application do not specifically limit the material of the film substrate.

[0048] 2. Explanation of the photosensitive unit array.

[0049] In this embodiment, the photosensitive unit array is disposed on a film substrate. When the film substrate is attached to the surface of the touch screen, the photosensitive unit array can cover the display area of ​​the touch screen. The photosensitive unit array can be used to collect the display light signal emitted by the touch screen.

[0050] The photosensitive unit array includes multiple photosensitive units, each capable of collecting light signals. When the film substrate is attached to the surface of the touchscreen, the multiple photosensitive units are distributed across the entire display area of ​​the touchscreen, allowing the photosensitive unit array to cover the display area. This application does not specifically limit the type of photosensitive unit; for example, the photosensitive unit can be a photosensor, a photodiode, etc.

[0051] 3. Explanation of the touch unit array.

[0052] In this embodiment, the touch unit array is disposed on a film substrate. When the film substrate is attached to the surface of the touch screen, the touch unit array can cover the display area of ​​the touch screen. The touch unit array can be used to output touch analog signals to the touch screen.

[0053] By outputting simulated touch signals to the touchscreen, touch actions can be simulated. These actions can include single click, double click, long press, swipe, drag, multi-finger zoom, multi-finger rotation, multi-finger tap, and handwriting input. For example, to achieve a swipe, the touch units located along the swipe path can be sequentially controlled to output touch signals. Similarly, to achieve a multi-finger zoom, two touch points can be formed using touch units, and these points can be controlled to move closer together (equivalent to shrinking) or further apart (equivalent to zooming in).

[0054] The touch unit array includes multiple touch units, each capable of outputting touch signals. When the film substrate is attached to the surface of the touch screen, the multiple touch units are distributed across the entire display area of ​​the touch screen, allowing the touch unit array to cover the display area. This application does not specifically limit the type of touch unit; for example, the touch unit can be a pressure touch unit, which can apply pressure to the touch screen to simulate touch operation, meaning the pressure signal is the touch signal output by the touch unit.

[0055] 4. Explain the control circuit.

[0056] In this embodiment of the application, the control circuit can be configured to perform the following steps S101 to S102.

[0057] Step S101: Generate a virtual screen for the touch screen based on the display light signal collected by the photosensitive unit array, identify the screen content of the virtual screen, and generate touch actions based on the screen content and the preset task script. The preset task script is used to indicate the automated operation process of the touch screen.

[0058] In some embodiments of this application, when generating a virtual screen for a touchscreen, an image can be generated based on a display light signal, and the virtual screen can be determined based on the image. The image generated based on the display light signal can intuitively display the screen content of the touchscreen; therefore, the screen content of the virtual screen obtained from this image can also represent the screen content of the touchscreen.

[0059] In some embodiments of this application, the automated operation process indicated by the preset task script may include multiple operation steps, execution conditions for each operation step, and execution actions. When generating touch actions based on the screen content and the preset task script, it is possible to identify whether the screen content meets the execution conditions of the operation step. If it is identified that the screen content meets the execution conditions of an operation step, then a touch action is generated based on the execution action of that operation step.

[0060] For example, one operation step is to click the login button on the login screen. The condition for this operation step is that the touchscreen displays the login screen, and the action is to click the login button. When the screen content is recognized as the login screen, the generated touch action is to click the login button.

[0061] Step S102: Control the touch unit array to output touch analog signals to the touch screen according to the touch action to execute the touch action, and identify the screen content of the virtual screen after the touch action is executed, and verify the execution result of the touch action according to the screen content.

[0062] Taking the example in step S101 as an example, if the touch action is to click the login button, when controlling the touch unit array to output touch simulation signals to the touch screen, the first position information of the login button in the virtual screen can be identified. According to the coordinate transformation relationship between the virtual screen and the touch screen, the first position information is transformed from the virtual screen coordinate system to the touch screen coordinate system to obtain the second position information of the login button on the touch screen. Then, according to the coordinate transformation relationship between the touch screen and the touch unit array, the second position information is transformed from the touch screen coordinate system to the coordinate system of the touch unit array to obtain the third position information of the login button on the touch unit array. Then, the touch unit in the touch unit array located at the third position information is controlled to output touch simulation signals that can simulate a click action to the touch screen, thereby realizing the click of the login button.

[0063] In some embodiments of this application, the automated operation process indicated by the preset task script may further include verification conditions for each operation step. When verifying the execution result of the touch action based on the screen content, it can be identified whether the screen content meets the verification conditions of the operation step. If it is identified that the screen content meets the verification conditions of the current operation step, it is determined that the touch action of the current operation step has been successfully executed. This application does not specifically limit the content of the verification conditions; for example, the verification condition may be that the screen content displays the content of a specified page.

[0064] Taking the example in step S101 as an example, when the operation step is to click the login button on the login interface, the verification condition can be that the touch screen displays a preset operation interface. After controlling the touch unit array to output touch simulation signals to the touch screen, if the screen content of the virtual screen is recognized to display the preset operation interface, then the verification condition is met and the touch action is successfully executed.

[0065] Based on the above structure, the intelligent device can acquire the screen content of the virtual screen through the photosensitive unit array and control circuit, thereby realizing the perception of the screen content of the touch screen. It then outputs touch simulation signals to the touch screen through the touch unit array, thus performing touch actions on the touch screen. Compared with existing technologies, the above intelligent device does not require calling the software interface of the electronic device (i.e., the electronic device equipped with the touch screen). Regardless of whether the electronic device has access to the software interface, it can still perceive and control the touch screen. Furthermore, the above intelligent device does not use robotic arms or other mechanisms to control the touch screen, reducing the complexity of the solution and maintenance costs. It also does not need to integrate photoelectric or touch sensors inside the touch screen, overcoming the limitation of not being able to automate the operation of the touch screen as an external accessory.

[0066] The following description continues regarding the intelligent device for touchscreen sensing and control provided in this application, specifically the membrane substrate.

[0067] See appendix Figure 2 In some embodiments of this application, the film-like substrate may include a screen coverage area and an edge functional area. The screen coverage area is used to cover the display area of ​​the touch screen, and the edge functional area is disposed at the edge of the screen coverage area. Specifically, the photosensitive unit array and the touch unit array are disposed in the screen coverage area so that when the film-like substrate is attached to the surface of the touch screen, the photosensitive unit array and the touch unit array can cover the display area of ​​the touch screen; the control circuit is disposed in the edge functional area.

[0068] Additionally, it should be noted that the edge functional area can be set on one or more edges of the screen coverage area. Figure 2 The edge functional area is shown as an example only, located on one edge of the screen coverage area.

[0069] In this embodiment, the smart device may further include functional modules such as a communication module, a battery module, and a storage module, all of which are also located in the edge functional area. The communication module is used to communicate with external devices and may include wired communication modules (such as Type-C interface-based communication modules or Ethernet communication modules) and / or wireless communication modules (such as Bluetooth communication modules or Wi-Fi communication modules). The battery module supplies power to the smart device. The storage module stores preset task scripts and the smart device's operating data (such as the execution results of touch actions in the aforementioned embodiments). Furthermore, when using a Type-C interface-based communication module, the Type-C interface can also be used to power the smart device. For example, Figure 1 The interface shown can be a Type-C interface, which can be used to power smart devices and communicate with external devices.

[0070] Based on the above embodiments, the modules in the smart device can be partitioned to ensure that the photosensitive unit array and the touch unit array can reliably sense and touch the touch screen, and to avoid interference from functional modules such as control circuits to the photosensitive unit array and the touch unit array, thereby further improving the reliability of sensing and touching the touch screen.

[0071] The following description continues with the smart device provided in this application, specifically the arrangement of the photosensitive unit array and the touch unit array in the screen coverage area.

[0072] In some embodiments of this application, the photosensitive unit array and the touch unit array can be arranged in any of the following ways to be located in the screen coverage area.

[0073] (1) The photosensitive unit array and the touch unit array are arranged in layers in the screen coverage area. For example, Figure 1 An example is shown showing a layered arrangement of the photosensitive unit array and the touch unit array.

[0074] (2) The photosensitive units in the photosensitive unit array and the touch units in the touch unit array are arranged alternately on the same plane within the screen coverage area. For example, the photosensitive units and touch units are represented as A and B respectively, and the arrangement of 5 photosensitive units and 5 touch units alternately can be represented as ABABABABAB.

[0075] (3) The screen coverage area is divided into multiple partitions, and multiple photosensitive units in a photosensitive unit array and multiple touch units in a touch unit array are respectively arranged in each partition. Within each partition, the photosensitive units and touch units are evenly distributed, so that the photosensitive units and touch units can cover each partition. Thus, when the film substrate is attached to the surface of the touch screen, the photosensitive units and touch units in each partition can reliably sense and touch each partition. Within each partition, the photosensitive units and touch units can be adjacent or not adjacent. This application embodiment does not specifically limit this, as long as the photosensitive units and touch units are evenly distributed within each partition and can reliably complete the sensing and touch functions.

[0076] Based on the above embodiments, various ways of setting up photosensitive unit arrays and touch unit arrays in the screen coverage area are provided, providing flexibility in setting up photosensitive unit arrays and touch unit arrays.

[0077] The following description continues with the intelligent device provided in this application, specifically the control circuit of the intelligent device.

[0078] In some embodiments of this application, when the control circuit verifies the execution result of a touch action based on the screen content, if the verification shows that the touch action failed, the touch action can be adjusted, and the touch unit array can be re-controlled to output a touch simulation signal to the touch screen based on the adjusted touch action. Taking the example in step S102, if the touch action is a click on the login button and the touch action fails, the third position information can be adjusted so that the adjusted third position information covers more touch units. Then, the touch units located at the third position information in the touch unit array are controlled to output a touch simulation signal that can simulate a click action to the touch screen. Alternatively, the duration and / or intensity of the touch unit outputting the touch simulation signal to the touch screen can be increased, i.e., the duration and / or intensity of clicking the login button can be increased. This application does not specifically limit the method of adjusting the touch action.

[0079] In some embodiments of this application, when the control circuit verifies the execution result of a touch action based on the screen content, if the verification shows that the touch action execution failed, it can re-identify the screen content of the virtual screen, regenerate the touch action based on the re-identified screen content and a preset task script, and control the touch unit array to output a touch analog signal to the touch screen based on the regenerated touch action. This embodiment can minimize the possibility of subsequent touch action execution failures due to incorrect screen content recognition.

[0080] In some embodiments of this application, when the control circuit verifies the execution result of a touch action based on the screen content, if the verification shows that the touch action execution has failed, it can execute a preset backup operation in the automated operation process. The preset task script may include preset backup operations for each operation step when the touch action execution fails. When the touch action execution failure of an operation step is detected, the backup operation of that operation step can be executed directly.

[0081] For example, a backup operation can ignore (or skip) the current operation step and send an exception handling request to the server. The server, based on the exception handling request, sends instructions to the electronic device equipped with the touchscreen. The electronic device, according to the instructions, controls the touchscreen to enter the next operation step of the automated operation process. The control circuit continues to sense and control the touchscreen according to the next operation step. Simultaneously, the server can record any failed touch actions in the current operation step so that after the entire automated operation process is completed, the user can quickly determine which operation steps failed based on the recorded information and manually verify these steps.

[0082] Based on the above embodiments, after verifying that the touch action has failed, efforts can be made to ensure that the touch action can be executed successfully, thus guaranteeing the smooth completion of the automated operation process of the touch screen.

[0083] The following description continues with the intelligent device provided in this application, focusing on the control circuit of the intelligent device.

[0084] In some embodiments of this application, the control circuit can also be configured to perform time-division multiplexing control on the photosensitive unit array and the touch unit array to avoid mutual interference between them. Specifically, within the photosensitive sampling time window, the photosensitive unit array can be controlled to collect the display light signal emitted by the touch screen, and the touch unit array can be shielded to prevent the touch unit array from interfering with the photosensitive unit array; within the touch execution time window, the touch unit array can be controlled to output touch analog signals to the touch screen, and the photosensitive unit array can be shielded to prevent the photosensitive unit array from interfering with the touch unit array.

[0085] The following description continues with the intelligent device provided in this application, focusing on the control circuit of the intelligent device.

[0086] In some embodiments of this application, the control circuit may also be configured to perform the following steps S201 to S203.

[0087] Step S201: Send the display light signal collected by the photosensitive unit array to the server.

[0088] Step S202: Receive touch commands sent by the server.

[0089] The server can generate a virtual screen based on the display light signal, identify the screen content of the virtual screen, generate touch actions based on the screen content of the virtual screen and a preset task script, and generate the aforementioned touch instructions based on the touch actions. The preset task script is the same as that in the previous embodiments. The methods by which the server generates the virtual screen, identifies the screen content of the virtual screen, and generates touch actions are the same as those in the previous embodiments, and will not be repeated here.

[0090] Step S203: Parse the touch command to obtain the touch action, control the touch unit array to output touch analog signals to the touch screen according to the touch action to execute the touch action, and send the execution completion information to the server after the touch action is executed.

[0091] The completion information is used to instruct the server to identify the screen content of the virtual screen and verify the execution result of the touch action based on the screen content. The method by which the server verifies the execution result of the touch action based on the screen content is the same as the method used by the control circuit in the aforementioned embodiment, and will not be described again here.

[0092] In some embodiments of this application, when multiple touch screens need to be automated, a smart device is used to sense and control each touch screen. The server can communicate with all smart devices and is responsible for generating, sensing, determining and verifying the virtual screens of all touch screens and the touch actions. Each smart device only needs to collect the display light signal of its corresponding touch screen and output touch simulation signals to its corresponding touch screen.

[0093] Based on the methods described in steps S201 to S203 above, the server can be responsible for the generation, perception, and determination and verification of touch actions of the virtual screen, which can reduce the computing power pressure on the control circuit in the smart device.

[0094] In some embodiments of this application, the control circuit can acquire information such as the currently executed operation steps, the currently executed touch actions, and the screen content of the virtual screen as interruption records and store them in the storage module of the smart device in abnormal situations such as power supply failure or operation interruption. When the abnormal situation returns to normal, the automated operation of the touch screen can continue according to the interruption records.

[0095] Similarly, in cases of power failure or operation interruption of a smart device, the server can obtain information such as the currently executed operation steps, the currently executed touch actions, and the screen content of the virtual screen as interruption records and store them in the server's memory. When the abnormal situation is resolved, the automated operation of the touch screen can be resumed based on the interruption records.

[0096] The technical solutions of this application have been described above with reference to the optional embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A smart device for touchscreen sensing and control, characterized in that, The smart device includes a film substrate and a photosensitive unit array, a touch unit array, and a control circuit disposed on the film substrate, wherein the control circuit is connected to the photosensitive unit array and the touch unit array respectively; The film-like substrate is used to attach to the surface of the touch screen; The photosensitive unit array is used to collect the display light signal emitted by the touch screen; The touch unit array is used to output touch analog signals to the touch screen; The control circuit is configured to perform the following operations: A virtual screen of the touch screen is generated based on the display light signal, the screen content of the virtual screen is identified, and touch actions are generated based on the screen content and a preset task script. The preset task script is used to indicate the automated operation process of the touch screen. The touch unit array is controlled to output the touch simulation signal to the touch screen according to the touch action to execute the touch action, and the screen content of the virtual screen is identified after the touch action is executed, and the execution result of the touch action is verified according to the screen content; The step of generating the virtual screen of the touch screen based on the display light signal includes: generating an image based on the display light signal, and determining the virtual screen based on the image.

2. The intelligent device according to claim 1, characterized in that, The film-like substrate includes a screen coverage area and an edge functional area. The screen coverage area is used to cover the display area of ​​the touch screen, and the edge functional area is disposed at the edge of the screen coverage area. The photosensitive unit array and the touch unit array are disposed in the screen coverage area; The control circuit is located in the edge functional area.

3. The intelligent device according to claim 2, characterized in that, The photosensitive unit array and the touch unit array are disposed in the screen coverage area, including: The photosensitive unit array and the touch unit array are arranged in layers in the screen coverage area.

4. The intelligent device according to claim 2, characterized in that, The photosensitive unit array and the touch unit array are disposed in the screen coverage area, including: The photosensitive units in the photosensitive unit array and the touch units in the touch unit array are arranged sequentially at intervals on the same plane within the screen coverage area.

5. The intelligent device according to claim 2, characterized in that, The photosensitive unit array and the touch unit array are disposed in the screen coverage area, including: The screen coverage area is divided into multiple partitions, and multiple photosensitive units in the photosensitive unit array and multiple touch units in the touch unit array are respectively set in each partition.

6. The intelligent device according to claim 1, characterized in that, The step of generating touch actions based on the screen content and a preset task script includes: The automated operation process indicated by the preset task script includes multiple operation steps, execution conditions for each operation step, and execution actions. If the screen content is found to meet the execution conditions of an operation step, the touch action is generated according to the execution action of the operation step.

7. The intelligent device according to claim 1, characterized in that, The step of verifying the execution result of the touch action based on the screen content includes: If the touch action fails, then one of the following will be executed: Adjust the touch action, and based on the adjusted touch action, re-control the touch unit array to output touch analog signals to the touch screen; The screen content of the virtual screen is re-identified, and a touch action is generated again based on the re-identified screen content and the preset task script. The touch unit array is then controlled to output a touch simulation signal to the touch screen based on the regenerated touch action.

8. The intelligent device according to claim 1, characterized in that, The control circuit is also configured to perform the following operations: Within the photosensitive sampling time window, the photosensitive unit array is controlled to collect the display light signal emitted by the touch screen, and the touch unit array is shielded. Within the touch execution time window, the touch unit array is controlled to output touch simulation signals to the touch screen, and the photosensitive unit array is shielded.

9. The intelligent device according to claim 1, characterized in that, The control circuit is also configured to perform the following operations: Send the display light signal collected by the photosensitive unit array to the server; Receive touch commands sent by the server; The touch command is parsed to obtain the touch action, and the touch unit array is controlled to output touch analog signals to the touch screen according to the touch action to execute the touch action. After the touch action is executed, execution completion information is sent to the server. The touch action is generated by the server based on the screen content of the virtual screen and a preset task script. The virtual screen is generated by the server based on the display light signal. The execution completion information is used to instruct the server to identify the screen content of the virtual screen and verify the execution result of the touch action based on the screen content.

Images