Playback methods, recording methods, and related devices

Abstract

This application discloses a playback method, a recording method, and related equipment, applied in the field of automation. It proposes adding time information to the semantics used to achieve automation as the basis for server scheduling and distributing multiple semantics. After obtaining multiple semantics, the server can distribute the semantics to the corresponding playback devices in chronological order according to the time information of each semantic, enabling each playback device to execute the semantics from the server in an orderly manner, thus achieving ordered playback. Since the server can replace manual delivery of semantics in a sequential manner, it can avoid confusion in the execution order of semantics by various playback devices in the system, thereby ensuring the automation process of a system containing multiple devices can be realized.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of automation, and in particular to a playback method, a recording method, and related devices. BACKGROUND

[0002] Automation technology is a technology that replaces repetitive operations of human beings by one or more devices. Generally, a workflow with regular driving, repeatability, and other characteristics can be a potential target of automation.

[0003] In the prior art, when a plurality of operations on a device requiring human intervention are automated, a user can copy semantics generated based on the plurality of operations to the device, and the device can implement corresponding operations based on the plurality of semantics, thereby achieving automation. When a system composed of a plurality of devices needs to be automated, the user needs to write semantics based on each device in the system respectively, and copy the semantics corresponding to each device to the corresponding device in the system respectively.

[0004] However, since the plurality of devices in the system are distributed, each device only perceives semantics input by the user to the device, and does not perceive the progress of the execution of semantics by other devices. If there is data or signaling interaction between the plurality of devices in the system (for example, after a device in the system executes a piece of semantics, another device in the system needs to execute another piece of semantics), the order of the execution of semantics by each device in the system will be confused, thereby making it difficult to achieve the automation process of the system. SUMMARY

[0005] Embodiments of the present application provide a playback method, a recording method, and related devices, which are used to implement an automation process of a system composed of a plurality of devices in a distributed scenario.

[0006] The playback method provided by the present application can be used in a scenario involving a recording device and a playback device, or in a scenario involving only a playback device without involving a recording device.

[0007] In a first aspect, the present application provides a playback method, which is used in a scenario involving a recording device and a playback device. In the method, each recording device in a recording group generates at least one semantic based on the operation of a user on the recording device. Then, a server receives at least one semantic from each recording device in the recording group. Each semantic is information describing the operation of the user on the recording device. The semantic includes time information, object information, and action information. The time information is used to indicate the time when the recording device detects the operation, which can reflect when the user operates the recording device. The object information is used to indicate the object of the operation corresponding to the semantic, i.e., which component on the recording device is operated by the user. The action information is used to indicate the content of the operation, i.e., which actions on the recording device are performed by the user. Then, the server sends each semantic to the corresponding playback device in the playback group in chronological order according to the time information of each semantic.

[0008] Optionally, when the server receives all the semantics generated by each recording device in the recording group, the server can obtain a plurality of semantics. In an example, the plurality of semantics can be a semantic set. In addition, the server needs to sort the semantics in the semantic set before distributing the semantics, so as to ensure that the semantics sent by the server to the playback group are complete, thereby facilitating the playback group to ensure that the operations performed by the playback devices based on the semantics are complete and accurate.

[0009] It should be understood that the at least one semantic of each recording device can be directly sent by the recording device to the server, i.e., the server receives the at least one semantic generated by each recording device from each recording device in the recording group; or the at least one semantic of each recording device can be sent by the recording device to a network element (e.g., a proxy network element, a switch, etc.) for message transfer, and then sent by the network element to the server. The specific implementation is not limited here.

[0010] In the embodiment, it is proposed to add time information in the semantic used to realize automation as a basis for the server to schedule and distribute a plurality of semantics. After obtaining the plurality of semantics, the server can distribute the plurality of semantics to the corresponding playback devices in chronological order according to the time information of each semantic in the plurality of semantics, so that each playback device can sequentially execute the semantics from the server, thereby realizing sequential playback. In the present application, since the server can replace manual implementation to sequentially distribute the plurality of semantics, the order of the playback devices in the system to execute the semantics can be avoided, thereby ensuring that the automation process of the system including a plurality of devices can be realized.

[0011] In the present application, the server needs to not only know the order in which each of the plurality of semantics is sent, but also determine which playback device in the playback group each of the plurality of semantics is sent to.

[0012] In a possible implementation, the server stores a first correspondence relationship between the indication information of each recording device in the recording group and the indication information of the playback device in the playback group, and the first correspondence relationship is used to indicate that the playback device can execute the semantics generated by the corresponding recording device. It can also be understood that the semantics generated by a recording device in the recording group can be run on the playback device in the playback group that has the first correspondence relationship with the recording device. Therefore, the server can determine which playback device in the playback group the semantics generated by each recording device in the recording group should be sent to based on the first correspondence relationship.

[0013] It should be understood that the first correspondence relationship can be preset by a person or determined by the server according to the information of the recording device (for example, the device type of the recording device) and the information of the playback device (for example, the device type of the playback device). It should be understood that the first correspondence relationship can be stored in the server in the form of an array, a table or other data structures capable of representing the association. The specific form is not limited here.

[0014] In the present embodiment, it is proposed that the server stores a first correspondence relationship between the indication information of each recording device in the recording group and the indication information of the playback device in the playback group. Therefore, the server can determine which playback device in the playback group the semantics generated by a recording device in the recording group should be sent to based on the first correspondence relationship. This not only helps to ensure the accuracy of the server when distributing semantics, but also helps to ensure the efficiency of the server when distributing semantics, avoiding the server distributing semantics to the wrong playback device and causing confusion in the playback task of the playback device.

[0015] In a possible implementation, the indication information in the first correspondence relationship includes a device identifier and / or a device type. For example, the indication information of the recording device includes a device identifier and / or a device type of the recording device; and the indication information of the playback device includes a device identifier and / or a device type of the playback device. Optionally, the indication information can also be other information capable of distinguishing a recording device or a playback device.

[0016] Optionally, the device type is used to indicate the software type and / or the hardware type supported by the recording device.

[0017] The hardware type refers to the physical structure of the device. For example, in a multi-terminal automation scenario based on a user interface (UI), the hardware type includes a touch screen, a keyboard, a mouse, and the like. For example, in a multi-terminal automation scenario involving video and audio pushing, the hardware type includes a camera, a speaker, a microphone, and a touch screen, and the like. For example, in a manufacturing-based automation scenario, the hardware type includes a distance sensor, an infrared sensor, an infrared thermometer, and the like. In addition, the software type refers to the software function that can be implemented by the device by running a program. For example, in an automation scenario based on a user interface (UI) application, the software type includes a system type recording which function interface is called by the application and a file type recording a UI storage format, the system type mainly includes a Web application calling a container interface such as a browser, a Windows desktop application calling a system interface, an Andriod / IOS / HarmonyOS native application, and the like; the file type recording a UI storage format mainly includes an application interface expressed based on a dom tree, an application interface supporting json format storage, an application interface supporting xml format storage, an application interface supporting pdf format storage, and the like. For example, in a manufacturing-based automation scenario, the software type includes a supported host computer type and a sensor-supported application type. For example, in an automation scenario based on a shortcut operation, the software type includes an application related to a certain business, such as a search engine application, a shopping application, and a video and audio playing application.

[0018] In a possible implementation, the software type supported by the recording device and the software type supported by the corresponding playback device have a non-empty intersection; and / or, the hardware type supported by the recording device and the hardware type supported by the corresponding playback device have a non-empty intersection. It can also be understood that the software type supported by the recording device and the playback device is completely the same or partially the same, or the hardware type supported by the recording device and the playback device is completely the same or partially the same.

[0019] In a possible implementation, each piece of the semantic includes the indication information of the recording device. The server sends the semantics to the corresponding playback device in the playback group in the time sequence according to the time information of each piece of the semantic, including: the server sends the semantics to the corresponding playback device in the playback group in the time sequence according to the time information of each piece of the semantic, the indication information of the recording device in each piece of the semantic, and the first correspondence.

[0020] In the embodiment, the indication information of the recording device is set in the semantic, so that the server determines which recording device in the recording group generates the semantic according to the indication information of the recording device in the semantic. Since the first correspondence relationship between the indication information of each recording device in the recording group and the indication information of the playback device in the playback group is also stored in the server, the server can determine the playback device corresponding to the semantic according to the indication information of the recording device in the semantic and the first correspondence relationship.

[0021] In a possible implementation, the server stores a device list, and the device list stores information of the recording group and information of the playback group. The information of the recording group includes the device identifier of each recording device in the recording group, and the information of the playback group includes the device identifier of each playback device in the playback group. Before the server receives at least one semantic from each recording device in the recording group, the method further includes: the server receives a plurality of registration messages, and each registration message includes the device identifier of the device sending the registration message; and the server determines whether the device is a recording device or a playback device according to the device list and the device identifier of the device in the registration message.

[0022] In the embodiment, the recording device and the playback device can provide the information of the recording device (for example, the device identifier of the recording device) and the information of the playback device (for example, the device identifier of the playback device) to the server through registration. The server needs to determine whether the device sending the registration request is a recording device or a playback device through the pre-stored device list. Through the foregoing registration management manner, the server can only manage the registered recording device and the playback device, which is beneficial to improving the efficiency of managing the recording device and the playback device.

[0023] In a possible implementation, the registration message further includes the device type of the device, and the device type of the device includes the device type of the recording device and the device type of the playback device. The server stores the device type of the recording device and the device identifier of the recording device correspondingly; and the server stores the device type of the playback device and the device identifier of the playback device correspondingly.

[0024] In the embodiment, when the device type (for example, the device type of the recording device or the device type of the playback device) is included in the registration message, the server also stores the device identifier and the device type of the device correspondingly, so that the server can conveniently check the device type of each device, and further facilitate the server to make decisions based on the device identifier and the device type.

[0025] In a possible implementation, after the server receives the plurality of registration messages, before the server receives at least one piece of semantics from each recording device in the recording group, the method further includes: the server sending a first recording instruction to each recording device in the recording group, the first recording instruction being used to instruct each recording device to start recording, the first recording instruction comprising system clock information of the server, the system clock information being used to instruct each recording device in the recording group to perform clock synchronization according to the system clock of the server.

[0026] In this embodiment, it is proposed that the server can control the recording devices to start recording through the first recording instruction, and the server can carry the system clock information of the server in the first recording instruction, so that the recording devices reset the system clock of the recording devices based on the system clock information of the server. Since the server broadcasts the first recording instruction to each recording device in the recording group at the same time, each recording device in the recording group can receive the aforementioned first recording instruction. Therefore, each recording device in the recording group can reset the clock based on the same system clock (i.e., the system clock of the server), so as to ensure that the system clocks of the recording devices in the recording group are synchronized.

[0027] Optionally, the first recording instruction further comprises at least one recording interface, the at least one recording interface being used to obtain original information of at least one operation corresponding to the task type of the recording task, the original information being information detected by the recording device and related to the operation of the user on the recording device. For example, the recording interface comprises an input / output (I / O) interface of a microphone, an I / O interface of a loudspeaker, an I / O interface of a mouse, an I / O interface of a keyboard, etc., and the specific implementation is not limited herein.

[0028] In this embodiment, it is proposed that the server can carry the recording interface in the first recording instruction, so that the recording devices obtain the original information only from the recording interface in the first recording instruction. This is beneficial to the server to control the recording devices to record based on the granularity of the control interface, without the recording devices to obtain information of each interface. This is beneficial to save the processing load of the recording devices.

[0029] In a possible implementation, after the server sends the first recording instruction to each recording device in the recording group, the method further includes: the server sending a second recording instruction to at least one recording device in the recording group, the second recording instruction being used to instruct the recording device to stop recording.

[0030] In the present application, when the server sends the semantics in the plurality of semantics (e.g., a set of semantics) to the corresponding playback device, the server can send one semantics to the corresponding playback device at a time, or send a group of semantics (including a plurality of semantics) to the corresponding playback device at a time. Optionally, the server can also receive a notification message from the playback device as a trigger condition for sending the next semantics (or the next group). The notification message is used to indicate that the playback device has executed the received semantics, and it can also be understood that the notification message is used to indicate that the playback device has successfully played back the received semantics.

[0031] In a possible implementation, if the plurality of semantics includes N semantics, the N is an integer greater than 1. The server sends the semantics in the plurality of semantics to the corresponding playback device in the playback group according to the time sequence information of each semantics, including: the server sorts the N semantics according to the time sequence information of each semantics in the N semantics to obtain a first semantic sequence determined by the N semantics; the server sends the i th semantics in the first semantic sequence to the corresponding playback device in the playback group; and the server receives a notification message corresponding to the i th semantics, the notification message corresponding to the i th semantics being used to indicate that the i th semantics is successfully played back, the i being an integer greater than or equal to 1 and less than or equal to N.

[0032] In the present embodiment, it is proposed that after the server sends one semantics to the playback device in the playback group each time, the server needs to receive a notification message from the playback device indicating that the playback is successful before triggering the sending of the next semantics. Therefore, it is beneficial for the playback devices in the playback group to be able to execute the received semantics in order, and it is also beneficial for the server to resend the semantics to the playback device when the playback device fails to send the notification message, so that each playback device in the playback group can execute the corresponding semantics, and thus ensure that the entire playback task is performed in order.

[0033] In a possible implementation, if the plurality of semantics includes N semantics, the N is an integer greater than 1. The server sends the semantics in the plurality of semantics to the corresponding playback device in the playback group according to the time sequence information of each semantics, including: the server sorts the N semantics according to the time sequence information of each semantics in the N semantics to obtain a first semantic sequence determined by the N semantics; the server divides the first semantic sequence into a plurality of second semantic sequences, each second semantic sequence being a sequence determined by at least one semantics in the first semantic sequence that is executed by the same playback device in sequence; and the server sends the second semantic sequence to the corresponding playback device in the playback group.

[0034] In this embodiment, the server sends one set of semantics (i.e., one second semantic sequence) to the playback devices in the playback group each time, and each set of semantics is executed by one playback device. After the playback device executes the aforementioned set of semantics, the server sends the next set of semantics to the playback device. This embodiment is advantageous in saving the signaling overhead between the server and the playback devices. In addition, since each semantic contains time information, the playback device can determine the order of executing each semantic in the aforementioned set of semantics based on the time information of each semantic in the set of semantics. Therefore, the playback devices in the playback group can also ensure that the entire playback task is performed in order.

[0035] In a second aspect, the present disclosure provides a playback method used in a scenario involving a recording device and a playback device. In this method, semantics can be manually copied to a server. The server obtains a plurality of semantics, each of which is used to indicate an operation that needs to be executed on a playback device in a playback group, and the operation includes controlling one playback device in the playback group to interact with another playback device in the playback group. The semantic includes timing information, object information, and action information. The timing information is used to indicate the order of the semantic relative to other semantics in the plurality of semantics. The object information is used to indicate the object of the operation corresponding to the semantic, i.e., which component of the playback device performs the operation according to the semantic. The action information is used to indicate the content of the operation, i.e., which actions are performed on the component of the playback device according to the semantic. Then, the server sends the semantic to the corresponding playback device in the playback group according to the order indicated by the timing information of each semantic.

[0036] In this embodiment, it is proposed to add timing information in the semantics used to realize automation as a basis for the server to schedule and distribute a plurality of semantics. After the server obtains the aforementioned plurality of semantics, the server can determine the order of distributing the semantics according to the timing information of each semantic in the aforementioned plurality of semantics, and then distribute the aforementioned plurality of semantics to the corresponding playback devices in the order indicated by the aforementioned timing information, so that each playback device can execute the semantics from the server in order, thereby realizing ordered playback. In this disclosure, since the server can replace manual distribution of semantics in order, it can avoid confusion in the order in which each playback device in the system executes the semantics, thereby ensuring that the automation process of a system containing multiple devices can be realized.

[0037] In a possible implementation, the server stores indication information of each playback device in the playback group, and each indication information of the playback device is used to identify one playback device in the playback group.

[0038] Optionally, the server can manage each playback device in the playback group in a registration manner. For example, each playback device in the playback group sends a registration message to the server to provide the playback device with the indication information of the playback device.

[0039] In a possible implementation, each piece of the semantics further includes the indication information of the playback device, and the indication information of the playback device in the semantics is used to indicate the playback device corresponding to the semantics.

[0040] In the embodiment, each piece of the semantics includes the indication information of the playback device, and the indication information of the playback device in the semantics is used to indicate which playback device will execute the semantics, so that the server distributes the semantics to the corresponding playback device based on the indication information of the playback device in each piece of the semantics.

[0041] In a possible implementation, the server sends the semantics to the corresponding playback device in the playback group according to the order indicated by the time sequence information of each piece of the semantics, including: the server sends the semantics to the playback device indicated by the indication information of the playback device according to the order indicated by the time sequence information of each piece of the semantics and the indication information of the playback device in each piece of the semantics.

[0042] In the embodiment, when the indication information of the playback device is included in each piece of the semantics, and the indication information of each playback device in the playback group is stored in the server, the server can determine the playback device corresponding to each piece of the semantics based on the indication information of the playback device in each piece of the semantics, and then the server can send the semantics to the corresponding playback device based on the indication information of the playback device in each piece of the semantics.

[0043] In a possible implementation, before the server sends the semantics to the corresponding playback device in the playback group according to the order indicated by the time sequence information of each piece of the semantics, the method further includes: the server obtains a second correspondence relationship, the second correspondence relationship including a correspondence relationship between the time sequence information and the indication information of the playback device, and the second correspondence relationship being used to indicate that the semantics including the time sequence information can run in the playback device indicated by the indication information of the playback device; and the server sends the semantics to the corresponding playback device in the playback group according to the order indicated by the time sequence information of each piece of the semantics, including: the server sends the semantics to the corresponding playback device in the playback group according to the order indicated by the time sequence information of each piece of the semantics and the second correspondence relationship.

[0044] In this embodiment, it is proposed that the server can determine which playback device each piece of semantics should be sent to based on the timing information of each piece of semantics. Specifically, the server can obtain a second correspondence relationship between the timing information and the indication information of the playback device. Therefore, the second correspondence relationship can indicate that the semantics containing the timing information should be run in the playback device indicated by the indication information of the playback device. Therefore, the server can determine the playback device corresponding to each piece of semantics based only on the timing information of the semantics and the second correspondence relationship, without parsing other information in the semantics.

[0045] In a possible implementation, the indication information of the playback device includes a device identifier of the playback device and / or a device type of the playback device, which is used to indicate a software type and / or a hardware type supported by the playback device. Optionally, the device type is used to indicate a software type and / or a hardware type supported by the playback device.

[0046] The hardware type refers to the physical structure of the device. For example, in a multi-terminal automation scenario based on a user interface (UI), the hardware type includes a touch screen, a keyboard, a mouse, and the like. For example, in a multi-terminal automation scenario involving video and audio pushing, the hardware type includes a camera, a speaker, a microphone, and a touch screen, and the like. For example, in an automation scenario based on manufacturing, the hardware type includes a distance sensor, an infrared sensor, an infrared thermometer, and the like. In addition, the software type refers to the software function that can be implemented by running a program. For example, in an automation scenario based on a user interface (UI) application, the software type includes a system type of a function interface called by a recording application and a file type of a UI storage format. The system type mainly includes a Web application calling a container interface such as a browser, a Windows desktop application calling a system interface, an Andriod / IOS / HarmonyOS native application; and the file type of the UI storage format mainly includes an application interface expressed based on a dom tree, an application interface supporting json format storage, an application interface supporting xml format storage, an application interface supporting pdf format storage, and the like. For example, in an automation scenario based on manufacturing, the software type includes a supported host computer type and a supported sensor application type. For example, in an automation scenario based on a shortcut operation, the software type includes a related application for performing a certain business, such as a search engine application, a shopping application, and a video and audio playing application.

[0047] In a third aspect, the present application provides a recording method applied to a recording device. Specifically, the recording device generates at least one semantic. Then, the recording device sends the at least one semantic to a server, so that the server sends the semantic to a corresponding playback device in a playback group in chronological order according to time information of each semantic.

[0048] wherein each semantic is information generated by the recording device and describing an operation of a user on the recording device, and the semantic comprises time information, object information and action information, the time information is used to indicate a time when the recording device detects the operation, the object information is used to indicate an object of the operation corresponding to the semantic, and the action information is used to indicate a content of the operation.

[0049] In this embodiment, it is proposed that the semantic generated by the recording device comprises time information, which is a time when the recording device detects the operation. When the semantic generated by each recording device in the recording group carries the time information, the aforementioned semantics from different recording devices can be sorted by the time information, so that the server can know the time sequence of the semantics generated by each recording device in the recording group. In the prior art, the semantic generated by the recording device does not have time information, and when the server obtains the semantics from different recording devices, the server cannot determine which semantics are generated first and which semantics are generated later. Therefore, the server cannot realize the ordered distribution of semantics. In the present application, the time information is added to the semantic, which can provide the server with a basis for sorting the semantics (i.e. the time information), and thus facilitate the server to realize the ordered distribution of semantics.

[0050] In a possible implementation, the recording device is one of a plurality of recording devices in the recording group, and the operation comprises controlling one recording device in the recording group to interact with another recording device in the recording group.

[0051] In this embodiment, it is proposed that each recording device in the recording group generates the semantic based on the aforementioned method, i.e. the semantic generated by each recording device in the recording group comprises the time information.

[0052] In a possible implementation, the recording device generates at least one semantic, including: the recording device acquires at least one original information, each of the original information being information detected by the recording device and related to an operation of a user on the recording device, each of the original information including an operation time, an operation object, and an operation type; the operation time is a time when the recording device detects the operation of the user on the recording device, the operation object is an object on which the user operates on the recording device detected by the recording device, and the operation type is a type of action performed by the user through hardware and / or software supported by the recording device detected by the recording device. The recording device determines the operation time as time information of the semantic; the recording device determines the operation object as object information of the semantic; and the recording device determines the operation type as action information of the semantic.

[0053] Optionally, the semantic further includes a device type of the recording device.

[0054] In a possible implementation, the recording device determines the action information of the semantic according to the operation type, including: the recording device determines, according to the operation type and a mapping rule, an action feature corresponding to the operation type as the action information of the semantic, the mapping rule including a corresponding relationship between an action feature and a plurality of operation types, each of the plurality of operation types satisfying the action feature.

[0055] In this embodiment, it is proposed that the recording device stores a mapping rule, which is used to convert the operation type in the original information into the action information in the semantic. The action information in the semantic is an action feature abstracted based on the operation type, and the action feature can be applied to a plurality of systems. This is beneficial to improve the universality of the semantic, so that the semantic can be executed in different systems.

[0056] For example, the plurality of operation types include a first operation type and a second operation type, the first operation type is pressing on a user interface through a touch screen, and the second operation type is pressing a left key on a display screen; a first action feature corresponding to the first operation type and the second operation type is pressing.

[0057] For example, the plurality of operation types include a third operation type and a fourth operation type, the third operation type is sliding in a first direction on a user interface through a touch screen, and the fourth operation type is scrolling a progress bar in the first direction through a mouse; a second action feature corresponding to the third operation type and the fourth operation type is moving in the first direction.

[0058] In a possible implementation, before the recording device generates the at least one semantic, the method further includes: the recording device sending, to the server, a registration message, the registration message including indication information of the recording device, the indication information of the recording device including a device type of the recording device and a device identifier of the recording device.

[0059] In a possible implementation, after the recording device sends the registration message to the server, the method further includes:

[0060] The recording device receives, from the server, a first recording instruction, the first recording instruction being used to instruct the recording device to start recording, the first recording instruction including system clock information of the server, the system clock information being used to instruct each recording device in the recording group to perform clock synchronization according to the system clock of the server.

[0061] In this embodiment, the recording device can receive the first recording instruction from the server and start recording based on the first recording instruction. In addition, since the system clock information of the server is carried in the first recording instruction, the recording device can reset the system clock of the recording device based on the system clock information of the server. When the server can broadcast the first recording instruction to each recording device in the recording group at the same time, each recording device in the recording group can receive the first recording instruction, and each recording device in the recording group can reset the clock based on the same system clock (i.e., the system clock of the server). Therefore, it is beneficial to ensure that the system clocks of the recording devices in the recording group are synchronized.

[0062] In a possible implementation, after the recording device receives the first recording instruction from the server, the method further includes: the recording device receiving, from the server, a second recording instruction, the second recording instruction being used to instruct the recording device to stop recording.

[0063] In a fourth aspect, the present application provides a server, when the server obtains semantics from a recording group, the server includes: a transceiver module and a processing module. Wherein, the transceiver module is used to receive at least one semantic from each recording device in the recording group, each semantic being information describing an operation of a user on the recording device, the semantic including time information, object information and action information, the time information being used to indicate a time when the recording device detects the operation, the object information being used to indicate an object of the operation corresponding to the semantic, and the action information being used to indicate a content of the operation. The processing module is used to determine a time sequence of each semantic according to the time information of each semantic, and control the transceiver module to send the semantic to a corresponding playback device in the playback group in the time sequence.

[0064] In a possible implementation, the server further includes a storage module configured to store a first correspondence relationship between the indication information of each recording device in the recording group and the indication information of the playback device in the playback group, the first correspondence relationship being used to indicate that the playback device can perform the semantics generated by the corresponding recording device.

[0065] Optionally, the indication information includes a device identifier and / or a device type, the device type being used to indicate a software type and / or a hardware type supported by the recording device.

[0066] Optionally, the software type supported by the recording device and the software type supported by the corresponding playback device have a non-empty intersection; and / or, the hardware type supported by the recording device and the hardware type supported by the corresponding playback device have a non-empty intersection.

[0067] In a possible implementation, when each piece of semantics includes the indication information of the recording device, the processing module is specifically configured to send, according to the time information of each piece of semantics, the indication information of the recording device in each piece of semantics and the first correspondence relationship, the semantics to the corresponding playback device in the playback group in the time sequence.

[0068] In a possible implementation, the storage module in the server is further configured to store a device list. The device list stores information of the recording group and information of the playback group. The information of the recording group includes a device identifier of each recording device in the recording group, and the information of the playback group includes a device identifier of each playback device in the playback group. At this time, the transceiver module in the server is further configured to receive a plurality of registration messages, each of the registration messages including a device identifier of a device sending the registration message; and the processing module in the server is further configured to determine, according to the device list and the device identifier of the device in the registration message, whether the device is a recording device or a playback device.

[0069] In a possible implementation, the registration message further includes a device type of the device, the device type of the device including a device type of the recording device and a device type of the playback device. When the registration message further includes the device type of the device, the storage module in the server is further configured to store the device type of the recording device in correspondence with the device identifier of the recording device; and store the device type of the playback device in correspondence with the device identifier of the playback device.

[0070] In a possible implementation, the transceiver module in the server is further configured to send a first recording instruction to each of the recording devices in the recording group, the first recording instruction being used to instruct each of the recording devices to start recording, the first recording instruction comprising system clock information of the server, the system clock information being used to instruct each of the recording devices in the recording group to perform clock synchronization according to the system clock of the server.

[0071] In a possible implementation, the transceiver module in the server is further configured to send a second recording instruction to at least one of the recording devices in the recording group, the second recording instruction being used to instruct the recording device to stop recording.

[0072] It should be noted that the embodiments of the present application have a variety of other implementations, and specific implementations and beneficial effects can be referred to the specific implementations of the first aspect and the beneficial effects, which will not be described here.

[0073] In a fifth aspect, the present application provides a server, which can directly obtain a plurality of semantics of artificial copying to obtain a semantic set. At this time, the server comprises an acquisition module and a processing module. The acquisition module is configured to acquire a plurality of semantics. The processing module is configured to control the transceiver module to send the semantics to the corresponding playback device in the playback group according to the order indicated by the time sequence information of each of the semantics.

[0074] Each of the semantics is used to indicate an operation that needs to be performed on the playback device in the playback group, and the operation comprises controlling one playback device in the playback group to interact with another playback device in the playback group. In addition, the semantics comprises time sequence information, object information and action information. The time sequence information is used to indicate the order of the semantics relative to other semantics in the plurality of semantics, the object information is used to indicate the object of the operation corresponding to the semantics, and the action information is used to indicate the content of the operation.

[0075] In a possible implementation, the server stores indication information of each of the playback devices in the playback group, and the indication information of each of the playback devices is used to identify one playback device in the playback group.

[0076] In a possible implementation, each of the semantics further comprises indication information of the playback device, and the indication information of the playback device in the semantics is used to indicate the playback device corresponding to the semantics.

[0077] In a possible implementation, the processing module is specifically configured to, according to the time sequence information of each of the semantics and the indication information of the playback device in each of the semantics, control the transceiver module to send the semantics to the playback device indicated by the indication information of the playback device according to the order indicated by the time sequence information.

[0078] In a possible implementation, the acquisition module is further configured to acquire a second correspondence relationship, the second correspondence relationship comprising a correspondence relationship between the timing information and the indication information of the playback device, and the second correspondence relationship is used to indicate that semantics containing the timing information can run in the playback device indicated by the indication information of the playback device. The processing module is further configured to determine the corresponding playback device of each piece of semantics according to the timing information of each piece of semantics and the second correspondence relationship, and control the transceiver module to send the semantics to the corresponding playback device in the playback group in the order indicated by the timing information.

[0079] In a possible implementation, the indication information of the playback device comprises a device identifier of the playback device and / or a device type of the playback device, and the device type of the playback device is used to indicate a software type and / or a hardware type supported by the playback device.

[0080] It should be noted that the embodiments of the present application have a variety of other implementations, and specific implementations and advantages thereof can be referred to the specific implementations and advantages of the second aspect, which will not be described here.

[0081] In a sixth aspect, the present application provides a recording device, comprising a processing module and a transceiver module. The processing module is configured to generate at least one piece of semantics, each piece of the semantics being information generated by the recording device and describing an operation of a user on the recording device, the semantics comprising time information, object information and action information, the time information being used to indicate a time when the recording device detects the operation, the object information being used to indicate an object of the operation corresponding to the semantics, and the action information being used to indicate a content of the operation. The transceiver module is configured to send the at least one piece of semantics to a server, so that the server sends the semantics to a corresponding playback device in a playback group in a time sequence according to the time information of each piece of the semantics.

[0082] In a possible implementation, the recording device is one of a plurality of recording devices in a recording group, and the operation comprises controlling one recording device in the recording group to interact with another recording device in the recording group.

[0083] Optionally, the semantics further comprises a device type of the recording device.

[0084] In a possible implementation, the processing module is specifically configured to:

[0085] obtaining at least one original information, each of the original information being information detected by the recording device and related to an operation of a user on the recording device, each of the original information including an operation time, an operation object and an operation type, wherein the operation time is a time when the recording device detects the operation of the user on the recording device, the operation object is an object operated by the user on the recording device detected by the recording device, and the operation type is a type of action performed by the user through hardware and / or software supported by the recording device detected by the recording device;

[0086] determining the operation time as time information of the semantic;

[0087] determining the operation object as object information of the semantic;

[0088] determining the operation type as action information of the semantic.

[0089] In a possible implementation, the processing module is specifically configured to determine, according to the operation type and a mapping rule, an action feature corresponding to the operation type as the action information of the semantic, the mapping rule including a corresponding relationship between one action feature and a plurality of operation types, and each of the plurality of operation types satisfies the action feature.

[0090] In a possible implementation, the transceiving module is further configured to send, to the server, a registration message including indication information of the recording device, the indication information of the recording device including a device type of the recording device and a device identifier of the recording device.

[0091] In a possible implementation, the transceiving module is further configured to receive, from the server, a first recording instruction, the first recording instruction being used to instruct the recording device to start recording, and the first recording instruction including system clock information of the server, the system clock information being used to instruct each recording device in the recording group to perform clock synchronization according to a system clock of the server.

[0092] In a possible implementation, the transceiving module is further configured to receive, from the server, a second recording instruction, the second recording instruction being used to instruct the recording device to stop recording.

[0093] It should be noted that the embodiments of the present application have a plurality of other implementations, and specific implementations and beneficial effects thereof can be referred to the specific implementations and beneficial effects of the third aspect, which will not be described here.

[0094] In a seventh aspect, the present application provides a server, comprising a processor and a memory; wherein the memory stores a computer program; and the processor invokes the computer program to enable the server to perform the method in the first aspect or any of the implementation forms of the first aspect, or to enable the server to perform the method in the second aspect or any of the implementation forms of the second aspect.

[0095] In an eighth aspect, the present application provides a recording device, comprising a processor and a memory; wherein the memory stores a computer program; and the processor invokes the computer program to enable the recording device to perform the method in the third aspect or any of the implementation forms of the third aspect.

[0096] In a ninth aspect, the present application provides an automation system, comprising: a playback device, the server in the fourth aspect and any of the implementation forms of the fourth aspect, and the recording device in the sixth aspect and any of the implementation forms of the sixth aspect.

[0097] In a tenth aspect, the present application provides an automation system, comprising: a playback device, and the server in the fourth aspect and any of the implementation forms of the fourth aspect.

[0098] In an eleventh aspect, the present application provides a computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method as described in the foregoing first aspect, second aspect or third aspect, and any of the implementation forms of the respective aspects.

[0099] In a twelfth aspect, the present application provides a computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method as described in the foregoing first aspect, second aspect or third aspect, and any of the implementation forms of the respective aspects. BRIEF DESCRIPTION OF DRAWINGS

[0100] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application.

[0101] FIG. 1A A system architecture diagram to which the method proposed by the present application is applicable;

[0102] FIG. 1B A scenario example diagram to which the method proposed by the present application is applicable;

[0103] FIG. 1C Another scenario example diagram to which the method proposed by the present application is applicable;

[0104] FIG. 1D Another example scenario for which the method presented in this application is applicable is shown in the figure below;

[0105] FIG. 2 A flowchart for the playback method and the recording method in this application is shown in the figure below;

[0106] FIG. 3 An example figure for the semantics in this application is shown in the figure below;

[0107] FIG. 4 Another flowchart for the playback method in this application is shown in the figure below;

[0108] FIG. 5 An example diagram for one embodiment of the server in this application is shown in the figure below;

[0109] FIG. 6 Another example diagram for another embodiment of the server in this application is shown in the figure below;

[0110] FIG. 7 An example diagram for one embodiment of the device in this application is shown in the figure below. DETAILED DESCRIPTION

[0111] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments of the present application.

[0112] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0113] For the sake of understanding, some terms related to the embodiments of the present application will be introduced first below:

[0114] Semantics: An executable file written in a specific descriptive language according to a certain format. This semantics can be recognized by a device, enabling the device to control its software and / or hardware based on the semantics, thereby causing the device to perform the operation corresponding to the semantics. Generally, one semantic corresponds to one operation. In some scenarios, semantics is also referred to as a script. In different embodiments of this application, the aforementioned semantics can be generated by a recording device or manually coded.

[0115] Recording refers to the process by which a device records user actions on the device using computer language (e.g., scripts, semantics, etc.). For example, the recording process includes: control search → execution and recording of the operation → generation of semantics. In this application, the device that generates semantics during the recording process is called a recording device, and a group of devices containing multiple recording devices is called a recording group.

[0116] Playback: refers to the process by which a device locates an object described by a computer language (e.g., script, semantics, etc.) and executes the corresponding operation. For example, the playback process includes: parsing semantics → locating controls → executing operations. In this application, the device that parses semantics in the playback process is called a playback device, and a group of devices containing multiple playback devices is called a playback group.

[0117] Multi-device collaboration: This refers to a scenario where multiple devices interact with each other to complete the same task. Collaboration can be software capability collaboration, hardware capability collaboration, or a combination of both.

[0118] Cross-device migration: Transferring semantics generated by one device to another device for execution, so that the other device can perform the operation corresponding to the aforementioned semantics.

[0119] The following section introduces the system architecture and application scenarios involved in the playback and recording methods proposed in this application:

[0120] like FIG. 1A The diagram shown illustrates the system architecture applicable to the playback and recording methods proposed in this application. The system includes a server 01 and a playback group 02. Server 01, a new structure added to the system in this application, is used to distribute semantics to each playback device (e.g., playback device 1 and playback device 2) in playback group 02. Each playback device (e.g., playback device 1 and playback device 2) in playback group 02 receives semantics from server 01 and executes the corresponding operation based on the semantics to collaboratively complete a task.

[0121] Optionally, the system can further include a recording group 03. The recording group 03 includes a plurality of recording devices (e.g., recording device 1 and recording device 2) for detecting operations on the recording devices and generating semantics based on the detected operations. As the aforementioned system adds the server 01, the recording devices in the recording group 03 can further send the generated semantics to the aforementioned server 01, so that the server 01 schedules the semantics generated by the recording devices. It should be understood that the number of recording devices in the recording group 03 is not necessarily the same as the number of playback devices in the playback group 02.

[0122] It should be understood that the devices (e.g., recording devices and playback devices) involved in the present application can be mobile terminals (MT) such as mobile phones, tablets, etc.; can be virtual reality (VR) terminal devices, augmented reality (AR) terminal devices; can be terminal devices in other fields that can have automation needs, such as terminal devices in industrial control, terminal devices in self-driving, terminal devices in remote medical surgery, terminal devices in smart grid, terminal devices in transportation safety, terminal devices in smart city, terminal devices in smart home, etc. It should be understood that the devices involved in the present application can also be other devices that can execute scripts or semantics, or other devices that have automation needs. The present application does not limit the specific technology and specific form of the devices.

[0123] In addition, the playback method and the recording method proposed in the present application are mainly applied to automation scenarios involving multiple devices.

[0124] For example, the aforementioned automation scenarios involving multiple devices can be automation scenarios based on human-computer interaction. For example, FIG. 1B As shown, it can be a scenario in which a mobile phone and a computer edit an email at the same time. In this scenario, the mobile phone 031 and the computer 032 are recording devices in the recording group 03. Among them, the mobile phone 031 converts the user's input of text or pictures into semantics and sends the semantics generated by the mobile phone 031 to the server 01, and the computer 032 converts the user's input of text in the email into semantics and sends the semantics generated by the computer 032 to the server 01. Then, the server 01 distributes the received semantics to the mobile phone 021 and the computer 022 in the playback group 02 for playback, so that the mobile phone 021 and the computer 022 can cooperate with each other to achieve email editing.

[0125] For example, the aforementioned automation scenario involving multiple devices can be a multi-terminal automation scenario involving audio-video pushing. For example, FIG. 1C As shown, the mobile phone 033 acts as a recording device in the recording group 03, which converts a series of operations such as video editing and debugging into semantics and reports the semantics to the server 01. Then, the server 01 distributes the received semantics to the computer 024 and the earphone 023 in the playback group 02 for playback, so that the computer 024 performs the semantics to realize the picture of the playback video, and the earphone 023 performs the semantics to realize the audio of the playback video.

[0126] For example, the aforementioned automation scenario involving multiple devices can be an automation scenario between multiple production devices in the manufacturing field. For example, FIG. 1D As shown, it can be a scenario in which one production device and another production device cooperate to manufacture a component. In this scenario, a user records the operations of two production devices (for example, production device A and production device B) in a virtual software through the computer 034, and reports the generated semantics to the server 01. Then, the server 01 distributes the semantics to the two production devices, so that production device A first performs a punching process on the component, then production device B performs a cutting process on the component, and then production device A further performs a punching process on the component. Further, the two production devices cooperate with each other to complete the manufacturing process.

[0127] It should be understood that the aforementioned FIG. 1B , FIG. 1C and FIG. 1D are only examples of automation scenarios involving multiple devices to which the method of the present application can be applied. In actual application, the aforementioned playback method and recording method can also be applied to a single-device automation scenario. For example, the recording group and the playback group each contain only one device, the server 01 collects semantics from the recording device in the recording group, and then the server distributes the semantics to the playback device in the playback group. Details are not described here.

[0128] As FIG. 2 shown, when the playback method proposed in the present application involves a recording device, the server, the recording device and the playback device will perform the following steps:

[0129] Step 201a, the recording device sends a registration message to the server; correspondingly, the server receives the registration message from the recording device.

[0130] Step 201b, the playback device sends a registration message to the server; correspondingly, the server receives the registration message from the playback device.

[0131] In this embodiment, the recording device can provide the information of the recording device to the server through the registration message, so that the recording device is registered to the server, and the server manages the recording device based on the information of the recording device. Similarly, the playback device can provide the information of the playback device to the server through the registration message, so that the playback device is registered to the server, and the server manages the playback device based on the information of the playback device.

[0132] Since the system in this embodiment includes both recording devices and playback devices, the registration message can be sent from a recording device in the recording group or a playback device in the playback group. Therefore, the server needs to determine whether the device sending the registration message is a recording device or a playback device according to the content of each registration message.

[0133] Specifically, the registration message includes the device identifier of the device sending the registration message. The device identifier can be an internet protocol (IP) address, a device factory number, or other information that can uniquely identify a device. Specifically, if the device sending the registration message is a recording device, the registration message carries the device identifier of the recording device; if the device sending the registration message is a playback device, the registration message carries the device identifier of the playback device. Therefore, if the server knows the device identifier of the recording device and the device identifier of the playback device, the server can determine whether the sender of the registration message is a recording device or a playback device based on the content carried by the registration message.

[0134] Optionally, the registration message further includes the device type of the device. Specifically, when the device sending the registration message is a recording device, the device type carried in the registration message is the device type of the recording device; when the device sending the registration message is a playback device, the device type carried in the registration message is the device type of the playback device.

[0135] The device type is used to indicate the hardware type and / or software type supported by the device.

[0136] The hardware type refers to the physical structure of the device. For example, in a multi-terminal automation scenario based on a user interface (UI), the hardware type includes a touch screen, a keyboard, a mouse, etc. For example, in a multi-terminal automation scenario involving video and audio pushing, the hardware type includes a camera, a speaker, a microphone, and a touch screen, etc. For example, in an automation scenario based on manufacturing, the hardware type includes a distance sensor, an infrared sensor, an infrared thermometer, etc.

[0137] In addition, the software type refers to a software function that the device can implement by running a program. For example, in a multi-terminal automation scenario based on a user interface (UI), the software type includes a system type that records which function interface an application calls and a file type that records a UI storage format. The system type mainly includes a web application that calls a container interface such as a browser, a Windows desktop application that calls a system interface, and an Andriod / IOS / HarmonyOS native application. The file type that records a UI storage format mainly includes an application interface expressed based on a dom tree, an application interface stored in a json format, an application interface stored in an xml format, and an application interface stored in a pdf format. For example, in an automation scenario based on manufacturing, the software type includes a supported host type and an application type supported by a sensor. For example, in an automation scenario based on a shortcut operation, the software type includes an application that performs a certain service, such as a search engine application, a shopping application, and a video and audio playing application.

[0138] It should be understood that for different application fields and different application scenarios, the specific implementation of the foregoing hardware type is different, and the specific implementation of the foregoing software type is also different. The specific implementation of the hardware type and the specific implementation of the software type are not limited in the present application.

[0139] It should be understood that in the present embodiment, the server can perform the foregoing steps 201a and 201b multiple times until the server receives the registration messages of all the recording devices in the recording group and the registration messages of all the playback devices in the playback group. In an implementation, the server can receive multiple registration messages in the same period, for example, the recording devices in the recording group and the playback devices in the playback group both send the registration messages to the server within a certain preset time range. In another implementation, the server can first receive the registration messages from all the recording devices in the recording group, and then receive the registration messages from all the playback devices in the playback group in a certain period.

[0140] In addition, whenever the server receives the foregoing registration message, the server will determine, according to the content of the registration message, whether the device that sends the registration message is a recording device or a playback device.

[0141] In one possible implementation, the server stores a device list, which stores information of a recording group and information of a playback group. The information of the recording group includes device identifiers of each recording device in the recording group, and the information of the playback group includes device identifiers of each playback device in the playback group. Whenever the server receives a registration message, the server compares the device identifier in the registration message with the device identifiers in the device list (i.e., the device identifiers of the recording devices or the device identifiers of the playback devices). When the device identifier in the registration message matches a device identifier of a recording device, the server determines that the device sending the registration message is a recording device. When the device identifier in the registration message matches a device identifier of a playback device, the server determines that the device sending the registration message is a playback device.

[0142] It should be appreciated that the device list includes device identifiers of each recording device in a recording group, and thus the server can determine not only whether the device sending a registration message is a recording device based on the registration message, but also whether all the recording devices in the recording group have registered with the server. Similarly, the device list includes device identifiers of each playback device in a playback group, and thus the server can determine not only whether the device sending a registration message is a playback device based on the registration message, but also whether all the playback devices in the playback group have registered with the server.

[0143] For example, the device list can be as shown in Table 1-1 below:

[0144] Table 1-1

[0145] Recording Group 1 LZ-01; LZ-02; LZ-03; Playback Group 1 HF-01; HF-02; HF-03;

[0146] In the device list shown in Table 1-1, the recording group 1 includes device identifiers of three recording devices, i.e., "LZ-01", "LZ-02" and "LZ-03", and the playback group 1 includes device identifiers of three playback devices, i.e., "HF-01", "HF-02" and "HF-03". If a registration message received by the server carries a device identifier "LZ-02", the server can determine that the device sending the registration message is a recording device in the recording group 1. When the server obtains "LZ-01", "LZ-02" and "LZ-03" from the three registration messages, the server can determine that all the recording devices in the recording group 1 have registered.

[0147] Optionally, when the registration message received by the server includes the device type of the device (e.g., the device type of the recording device and the device type of the playback device), the server can store the device type in the device list in correspondence, so as to facilitate subsequent processes to consult the device type of the recording device and / or the device type of the playback device. Specifically, the server can store the device type of the recording device in correspondence with the device identifier of the recording device, and store the device type of the playback device in correspondence with the device identifier of the playback device.

[0148] For example, in a multi-terminal automated scenario based on human-computer interaction, the device list can be as shown in Table 1-2:

[0149] Table 1-2

[0150]

[0151] In step 202, the server obtains a first correspondence relationship, which is used to indicate that the playback device can execute the semantics generated by the corresponding recording device.

[0152] In this embodiment, the first correspondence relationship is the correspondence relationship between the indication information of each recording device in the recording group and the indication information of the playback device in the playback group. The indication information includes the device identifier and / or the device type. Optionally, the indication information can also be other information capable of distinguishing different recording devices, and other information capable of distinguishing different playback devices.

[0153] In one possible implementation, the foregoing indication information is the device identifier, i.e., the indication information of the recording device in the first correspondence relationship is the device identifier of the recording device, and the indication information of the playback device in the first correspondence relationship is the device identifier of the playback device. Taking the device identifier in the foregoing Table 1-1 as an example, the first correspondence relationship includes: “LZ-01” corresponds to “HF-02”; “LZ-02” corresponds to “HF-03”; and “LZ-03” corresponds to “HF-01”.

[0154] This embodiment can be applied to a scenario in which the device type of the recording device in the recording group is the same as or similar to the device type of the playback device in the playback group. At this time, using the device identifier of each device to represent the first correspondence relationship is beneficial to ensuring that the recording device and the playback device are clearly corresponding, and avoiding confusion.

[0155] In another possible implementation, the aforementioned indication information is a device type, i.e., the indication information of the recording device in the first correspondence relationship is a device type of the recording device, and the indication information of the playback device in the first correspondence relationship is a device type of the playback device. Taking the device types in the aforementioned Table 1-2 as an example, the first correspondence relationship includes: "recording device supporting Windows" corresponds to "playback device supporting Windows"; "recording device supporting Android" corresponds to "playback device supporting Android"; and "recording device supporting IOS" corresponds to "playback device supporting IOS". For another example, the first correspondence relationship includes: "recording device supporting IOS" corresponds to "playback device supporting Windows"; "recording device supporting Android" corresponds to "playback device supporting Android"; and "recording device supporting Windows" corresponds to "playback device supporting IOS". For yet another example, the first correspondence relationship includes: "recording device supporting keyboard and mouse" corresponds to "playback device supporting keyboard and mouse"; "recording device supporting touch screen" corresponds to "playback device supporting touch screen"; and "recording device supporting speaker" corresponds to "playback device supporting speaker". In actual applications, there are other examples of using device types to represent the first correspondence relationship, which are not described herein.

[0156] In actual applications, the first correspondence relationship can also be represented by at least one hardware type and / or at least one software type in the device types. For example, the first correspondence relationship includes: "recording device supporting keyboard, mouse, and xml format" corresponds to "playback device supporting keyboard, mouse, and xml format"; "recording device supporting speaker and Android system" corresponds to "playback device supporting speaker and IOS system"; and "recording device supporting touch screen and Android system" corresponds to "playback device supporting touch screen and Android system". Specific examples are not listed herein.

[0157] It should be understood that the first correspondence relationship can be stored in the server in the form of an array, a table, or other data structure capable of representing an association. Specific examples are not limited herein.

[0158] In this embodiment, the aforementioned first correspondence relationship can be preset by a person or determined by the server according to the device type of the recording device and the device type of the playback device. The two cases are described as follows:

[0159] In a possible implementation, the first correspondence relationship is a preset correspondence relationship. At this time, the step of obtaining the first correspondence relationship by the server can be understood as that the server reads the aforementioned first correspondence relationship from a storage medium, or the server obtains the first correspondence relationship set by a person through an input / output (I / O) interface.

[0160] In another possible implementation, the first correspondence relationship is determined by the server based on the device type of the recording device and the device type of the playback device. Specifically, the server can obtain the device identifier and the device type of each recording device from the registration message in step 201a, and obtain the device identifier and the device type of each playback device from the registration message in step 201b, and then obtain a device list containing the device identifier of the recording device, the device type of the recording device, the device identifier of the playback device, and the device type of the playback device. For example, the device list shown in Tables 1-2.

[0161] In an implementation, the server can determine the recording device and the playback device supporting the same device type as having the first correspondence relationship. It can also be understood that the software type supported by the recording device and the software type supported by the corresponding playback device (i.e., the playback device having the first correspondence relationship with the recording device) have a non-empty intersection; and / or the hardware type supported by the recording device and the hardware type supported by the corresponding playback device (i.e., the playback device having the first correspondence relationship with the recording device) have a non-empty intersection. For example, the server determines that the recording device supporting the Windows system in the recording group and the playback device supporting the Windows system in the playback group have the first correspondence relationship. For another example, the server determines that the recording device supporting the camera in the recording group and the playback device supporting the camera in the playback group have the first correspondence relationship. For another example, the server determines that the recording device supporting the speaker in the recording group and the playback device supporting the speaker in the playback group have the first correspondence relationship. The specific implementation is not limited here.

[0162] For example, the first correspondence relationship can be as shown in Table 2-1:

[0163] Table 2-1

[0164]

[0165] In the example shown in Table 2-1, the recording devices and the playback devices in the same row have the first correspondence. Taking the second row in Table 2-1 as an example, the recording device LZ-01 in the recording group supporting Windows system and supporting keyboard and mouse has the first correspondence with the recording device HF-01 in the playback group supporting Windows system and supporting keyboard and mouse. Taking the third row in Table 2-1 as an example, the recording device LZ-02 in the recording group supporting Android system and supporting touch screen has the first correspondence with the recording device HF-02 in the playback group supporting Android system and supporting touch screen. Similarly, the specific examples are not listed here.

[0166] Optionally, if the server stores a task list recording the recording tasks of the recording devices, the aforementioned non-empty intersection is related to the recording task of the recording device. It can also be understood that the software type supported by the recording device and the software type supported by the playback device are both related to the recording task; and / or, the hardware type supported by the recording device and the hardware type supported by the playback device are both related to the recording task.

[0167] The task list includes the task information of each recording device in the recording group, and the task information includes the task type of the recording device and the hardware type and / or software type required by the recording device to support the task type. Generally, different task types require the device to support different software types and / or hardware types. Specifically, the server will determine the recording device and the playback device having the first correspondence based on the task type of each task and the hardware type and / or software type required by the recording device to support the task type. For example, for tasks involving taking pictures, recording videos, etc., the recording device and the playback device are required to support the camera. For example, for tasks involving audio playback, the recording device and the playback device are required to support the speaker. For example, for tasks involving voice calls, the recording device and the playback device are required to support the microphone. The specific examples are not listed here.

[0168] It should be noted that the device type of the recording device and the device type of the playback device having the first correspondence are not necessarily completely the same. For example, for tasks involving taking pictures, recording videos, etc., the recording device and the playback device are required to support the camera, but it is not necessarily required that the recording device and the playback device support the same system type. At this time, it can be determined that the recording device supporting the Android system and supporting the camera has the first correspondence with the playback device supporting the IOS system and supporting the camera. For example, for tasks involving email editing, the recording device and the playback device are required to support a certain mailbox application, but it is not necessarily required that the recording device and the playback device support touch screen input. At this time, it can be determined that a mobile phone supporting the aforementioned mailbox application and supporting touch screen input is used as the recording device, and a computer supporting the aforementioned mailbox application and supporting keyboard input is used as the playback device.

[0169] It should be understood that the step 201a, the step 201b and the step 202 can all be performed before the step 203, i.e. before recording, the recording device and the playback device are both registered to the server, and the server obtains the first correspondence. The step 201b and the step 202 can also be performed after the step 203 and before the step 206, i.e. the recording device in the recording group is registered to the server first, and records under the instruction control of the server, while the playback device in the playback group only needs to be registered to the server before the server distributes semantics.

[0170] The step 203, the server sends a first recording instruction to each recording device in the recording group; correspondingly, the recording device receives the first recording instruction from the server.

[0171] The first recording instruction is used to instruct each recording device to start recording. When the recording device receives the foregoing first recording instruction, the recording device will start recording. Wherein, starting recording refers to the process that the recording device starts to detect the operation of the user on the recording device, and generates semantics based on the operation detected by the recording device.

[0172] Optionally, the first recording instruction includes system clock information of the server, and the system clock information is used to instruct each recording device in the recording group to perform clock synchronization according to the system clock of the server. Specifically, after each recording device receives the system clock information in the foregoing first recording instruction, the recording device can reset the system clock. This embodiment is advantageous to ensure that the system clocks of the various recording devices in the recording group are synchronized.

[0173] Optionally, when the server stores a task list, and the task information in the task list includes a recording interface corresponding to each task, the foregoing first recording instruction further includes at least one recording interface, and the at least one recording interface is related to the foregoing task. Exemplarily, the foregoing recording interface includes an I / O interface of a microphone, an I / O interface of a loudspeaker, an I / O interface of a mouse, an I / O interface of a keyboard, etc., which are not limited here.

[0174] The step 204, the recording device generates at least one piece of semantics.

[0175] The semantic is information generated by the recording device to describe the operation of the user on the recording device. It can also be understood that the semantic is an abstract expression of the recording device to the operation of the user on the recording device. The semantic includes time information, object information and action information. The time information is used to indicate the time when the recording device detects the operation, which can reflect when the user operates the recording device. The object information is used to indicate the object of the operation corresponding to the semantic, that is, which component on the recording device is operated by the user. The action information is used to indicate the content of the operation, that is, which actions on the recording device are performed by the user.

[0176] In the embodiment, after the recording device receives the first recording instruction, the recording device will start recording and generate semantics. Specifically, the recording device will detect the operation of the user on the recording device through the recording interface, obtain the original information corresponding to the operation, and generate a semantic based on one or more original information. It should be understood that if the first recording instruction includes at least one recording interface, the recording device will only detect the operation of each interface in the at least one recording interface when recording. For example, if the first recording instruction includes the I / O interface of the mouse and the I / O interface of the keyboard, but does not include the I / O interface of the microphone and the I / O interface of the speaker, the recording device will only obtain information related to the operation of the user from the I / O interface of the mouse and the I / O interface of the keyboard when recording, and will not obtain information from the I / O interface of the microphone and the I / O interface of the speaker.

[0177] In addition, each semantic includes indication information of the recording device, which is used to indicate which recording device generates the semantic, so that the server distributes the semantic to the corresponding playback device based on the indication information of the recording device in each semantic.

[0178] In the embodiment, when the recording device detects the operation of the user, it can obtain the original information corresponding to the operation, and then generate a semantic based on the original information. Each original information is information detected by the recording device related to the operation of the user on the recording device. Each original information includes operation time, operation object and operation type. The operation time is the time when the recording device detects the operation of the user on the recording device; the operation object is the object of the operation of the user on the recording device detected by the recording device; and the operation type is the type of action performed by the user through the hardware and / or software supported by the recording device. Then, the recording device determines the operation time as the time information of the semantic. In addition, the recording device determines the object information of the semantic according to the operation object. At the same time, the recording device determines the action information of the semantic according to the operation type.

[0179] The recording device stores a mapping rule for determining the action information, and the mapping rule includes a corresponding relationship between an action feature and a plurality of operation types. Each of the plurality of operation types satisfies a corresponding action feature. Specifically, the recording device determines the action feature corresponding to the operation type as semantic action information according to the operation type and the mapping rule. For example, if the plurality of operation types in the mapping rule include a first operation type and a second operation type, and the first operation type is pressing on a user interface through a touch screen, and the second operation type is pressing the left button on a display screen through a mouse, the first action feature corresponding to the first operation type and the second operation type is pressing. For example, if the plurality of operation types in the mapping rule include a third operation type and a fourth operation type, and the third operation type is sliding in a first direction on a user interface through a touch screen, and the fourth operation type is rolling a progress bar in the first direction through a mouse, the second action feature corresponding to the third operation type and the fourth operation type is moving in the first direction.

[0180] For ease of understanding, a multi-terminal automation scenario based on a user interface (UI) is taken as an example. If the recording device detects that a user applies an action of pressing at point A with coordinates (x, y) on the touch screen through the I / O interface of the touch screen, and the current system time is T1, the recording device will obtain the original information {T1; A(x, y); pressing on the touch screen}. Wherein, "T1" is the operation time, "A(x, y)" is the operation object, and "pressing on the touch screen" is the operation type. At this time, the recording device will determine "T1" as semantic time information. At the same time, the recording device will determine the element or picture at the position of coordinates A(x, y) as object information. In addition, the recording device will also determine the action information based on the operation type and the mapping rule.

[0181] For example, if the mapping rule stored in the recording device at this time is shown in Table 3-1, the recording device will determine the action information as "pressing".

[0182] Table 3-1

[0183]

[0184] For example, in a multi-terminal automation scenario based on a user interface (UI), for example, a UI-based front-end test scenario. Some semantics generated by the recording device can be as follows: FIG. 3As shown in the diagram. "end_id: LZ-01" indicates that the recording device's device identifier is "LZ-01". "end_type: Android, file_type: XML" indicates the device type of the recording device; the recording task of this device needs to support the "Android" system type and the "XML" file type. "time: T1" indicates the time information, meaning the recording device detected the user's operation on the recording device at time T1. "object_type: element; object_value: {name: name_string; class: class_string; path: element_path_string}" represents object information. "object_type: element" indicates that the object information is a specific element in the user interface; "object_value: {name: name_string; class: class_string; path: element_path_string}" indicates that the aforementioned element was found in the user interface, which can also be understood as representing the characteristics of the aforementioned element. Here, "name: name_string" indicates the element's name in the system as "name_string", "class: class_string" indicates the element's level in the system as "class_string", and "path: element_path_string" indicates the element's search path in the system as "path: element_path_string". "action_type: point_down; action_value" represents the action information, where "point_down" indicates the action is a press, and "action_value" records other information related to the aforementioned action.

[0185] For example, if a Backus normal form (BNF) is defined to represent the semantics generated by the recording device, then the semantics generated by the recording device can be represented by the paradigms and meanings listed in Table 4-1 below.

[0186] Table 4-1

[0187]

[0188] It should be understood that the preceding text FIG. 3 The examples listed in Table 4-1 are merely one of many examples in this embodiment. In practical applications, other methods can also be used to represent the aforementioned semantics. Specifically, this embodiment does not limit the specific implementation of the semantics.

[0189] In this embodiment, the recording device generates semantics based on the detected original information until the recording device receives a second recording instruction from the server, the second recording instruction being used to instruct the recording device to stop recording; or until the recording device completes all operations in the recording task.

[0190] In step 205, the recording device sends the aforementioned at least one semantic to the server; correspondingly, the server receives the aforementioned at least one semantic from the recording device.

[0191] In this embodiment, after the recording device generates semantics, the recording device needs to send the semantics to the server. Specifically, the recording device can send each semantic to the server as it is generated; or the recording device can send all semantics generated by the recording device to the server after stopping recording, which is not limited here.

[0192] It should be understood that the recording device sending semantics to the server can be the recording device sending a data packet carrying semantics to the server, or the recording device sending a data packet carrying semantics to a network element (for example, a proxy network element, a switch, etc.) that relays messages, and then sending the data packet to the server by the network element. This is not limited here.

[0193] It should be understood that the server will receive at least one semantic from each recording device in the recording group, and will obtain multiple semantics. For example, the aforementioned multiple semantics can constitute a semantic set, which includes all semantics generated by each recording device in the recording group. For ease of introduction, the semantic set is used as an example in the following.

[0194] In step 206, the server sends semantics to the corresponding playback device in the playback group according to the time information of each semantic in chronological order.

[0195] In this step, the server first determines the corresponding playback device of each semantic (i.e., which playback device in the playback group should each semantic be sent to); then the server analyzes the time information of each semantic, sorts all semantics in the semantic set in chronological order; then the server sends the semantics in the semantic set to the corresponding playback device in the playback group in chronological order.

[0196] The server can determine the playback device corresponding to each semantic according to the first correspondence relationship introduced above. Since each semantic contains the indication information of the recording device, and the first correspondence relationship stores the correspondence relationship between the indication information of each recording device in the recording group and the indication information of the playback device in the playback group, the server can determine the playback device corresponding to the semantic based on the indication information of the recording device and the first correspondence relationship. For ease of understanding, taking the first correspondence relationship as Table 2-1 in the foregoing description as an example, if the server determines that the indication information of the recording device carried in semantic 01 is "LZ-01", the server can determine that the playback device corresponding to semantic 01 is the playback device with indication information "HF-01".

[0197] In addition, when the server sends the semantics in the semantic set to the corresponding playback device, the server can send only one semantic to the corresponding playback device at a time, or can send a group of semantics (including multiple semantics) to a certain playback device at a time. Optionally, the server can also use the notification message received from the playback device as a trigger condition for sending the next semantic (or the next group). The notification message is used to indicate that the playback device has executed the received semantic, and can also be understood as indicating that the semantic received by the playback device has been successfully played back.

[0198] In a possible implementation, the semantic set includes N semantics, and N is an integer greater than 1. The server sorts the semantics in the semantic set according to the time sequence information of the N semantics in the semantic set, to obtain a first semantic sequence determined by the N semantics. Then, the server sends the i-th semantic in the first semantic sequence to the corresponding playback device in the playback group (where i is an integer greater than or equal to 1 and less than or equal to N). Then, the server receives the notification message corresponding to the i-th semantic, which is used to indicate that the i-th semantic has been successfully played back.

[0199] In this embodiment, it is proposed that after the server sends one semantic to the playback device in the playback group each time, the server needs to receive the notification message indicating successful playback from the playback device before triggering the sending of the next semantic. Therefore, it is beneficial for the playback device in the playback group to be able to execute the received semantics in order, and it is also beneficial for the server to resend the semantic to the playback device when the playback device fails to send the notification message, so that each playback device in the playback group can execute the corresponding semantic, thereby ensuring that the entire playback task is performed in order.

[0200] In another possible implementation, the semantic set includes N semantics, where N is an integer greater than 1. The server orders the semantics in the semantic set according to the time sequence information of the N semantics in the semantic set, to obtain a first semantic sequence determined by the N semantics. Then, the server divides the first semantic sequence into a plurality of second semantic sequences, each of which is a sequence of at least one semantic in the first semantic sequence that is executed by the same playback device in sequence. Then, the server sends the second semantic sequences to the corresponding playback devices in the playback group in sequence until the server sends all the second semantic sequences determined based on the aforementioned semantic set to the playback devices.

[0201] In this embodiment, the server sends a group of semantics (i.e., one second semantic sequence) to the playback devices in the playback group at a time, and the group of semantics are executed by a certain playback device. After the playback devices execute the aforementioned group of semantics, the server sends the next group of semantics to the playback devices. This embodiment is advantageous in saving the signaling overhead between the server and the playback devices. In addition, since each semantic includes time information, the playback devices can determine the order of executing each semantic in the group of semantics based on the time information of each semantic in the group of semantics. Therefore, the playback devices in the playback group can also ensure that the entire playback task is performed in order.

[0202] In this embodiment, time information is added to the semantics for automation, which is used by the server to schedule and distribute a plurality of semantics. After the server obtains the plurality of semantics (e.g., a semantic set), the server can distribute the plurality of semantics to the corresponding playback devices in time sequence according to the time information of each semantic in the plurality of semantics, so that each playback device can execute the semantics from the server in order, thereby realizing ordered playback. In this application, since the server can replace manual distribution of the plurality of semantics in order, the order of executing the semantics by each playback device in the system can be avoided, and thus the automation process of the system including a plurality of devices can be realized.

[0203] As shown in FIG. 4, when the playback method proposed in this application only involves the playback devices, the server and the playback devices will perform the following steps: FIG. 4

[0204] Step 401, the server obtains a plurality of semantics.

[0205] The plurality of semantics can be directly input into the server by the user through an I / O interface, or can be read from an external storage medium by the server, which is not limited here. For example, the plurality of semantics can constitute a semantic set. For ease of introduction, a semantic set is used as an example in the following description.

[0206] ​Each of the semantics is used to indicate an operation that needs to be performed on the playback devices in the playback group, and the operation includes controlling one of the playback devices in the playback group to interact with another of the playback devices in the playback group.

[0207] The semantic includes timing information, object information and action information. The timing information is used to indicate the order of the semantic relative to other semantics in the semantic set. For example, the aforementioned timing information can be Arabic numerals such as "1", "2", "3", etc. that can represent the order. The object information is used to indicate the object of the operation corresponding to the semantic, i.e. which component of the playback device performs the operation according to the semantic. The action information is used to indicate the content of the operation, i.e. which actions are performed on the component of the playback device according to the semantic.

[0208] Optionally, each of the aforementioned semantics includes indication information of the playback device, and the indication information of the playback device in the semantic is used to indicate which playback device performs the semantic, so as to facilitate the server to distribute the semantic to the corresponding playback device based on the indication information of the playback device in each semantic.

[0209] The indication information of the playback device includes a device identifier of the playback device and / or a device type of the playback device.

[0210] In a possible implementation, the indication information of the playback device is the device identifier of the playback device. At this time, the device identifier of the playback device can be an IP address of the playback device, a factory number of the playback device, etc. that can uniquely identify one playback device.

[0211] In another possible implementation, the indication information of the playback device is a device type of the playback device. The device type of the playback device is used to indicate a software type and / or a hardware type supported by the playback device. The hardware type refers to a physical structure of the device. For example, in a multi-terminal automation scenario based on a user interface (UI), the hardware type includes a touch screen, a keyboard, a mouse, and the like. For example, in a multi-terminal automation scenario related to audio and video pushing, the hardware type includes a camera, a speaker, a microphone, and a touch screen, and the like. For example, in an automation scenario based on manufacturing, the hardware type includes a distance sensor, an infrared sensor, an infrared thermometer, and the like. In addition, the software type refers to a software function that can be implemented by the device by running a program. For example, in a multi-terminal automation scenario based on a user interface (UI), the software type includes a system type of recording an application calling a function interface and a file type of a UI storage format. The system type mainly includes a Web application calling a container interface such as a browser, a Windows desktop application calling a system interface, an Andriod / IOS / HarmonyOS native application; and the file type of the UI storage format mainly includes an application interface expressed based on a dom tree, an application interface supporting json format storage, an application interface supporting xml format storage, an application interface supporting pdf format storage, and the like. For details, refer to the description of the device type in steps 201a and 201b, which will not be repeated here.

[0212] It should be understood that for different application fields and different application scenarios, the specific implementation of the foregoing hardware type is different, and the specific implementation of the foregoing software type is also different. The specific implementation of the hardware type and the specific implementation of the software type are not limited in the present application.

[0213] In addition, the representation of the semantics in the present embodiment can refer to the examples introduced in the foregoing FIG. 3 and Table 4-1, which will not be repeated here.

[0214] In step 402, the server obtains information of the playback device.

[0215] In the present embodiment, the server can obtain the information of the playback device by receiving a registration message from the playback device. Specifically, the server receives one or more registration messages, each of which contains the device identification, the device type, and the like of the playback device. Then, the server stores the device identification and the device type of each playback device, so that the server can obtain the information of each playback device in the subsequent distribution of semantics.

[0216] In step 403, the server sends the semantics to the corresponding playback device in the playback group according to the order indicated by the time sequence information of each semantics.

[0217] In this step, the server will first determine the playback device corresponding to each semantic (i.e., which playback device in the playback group should each semantic be sent to); then, the server parses the timing information of each semantic and sends the semantics in the semantic set to the corresponding playback device in the playback group in the order indicated by the timing information.

[0218] In this embodiment, the server has multiple ways to determine the playback device corresponding to each semantic, which will be introduced as follows:

[0219] In one possible implementation, each semantic includes indication information of a playback device, which is used to indicate which playback device the semantic will be executed by. Therefore, the server will parse the received semantics to obtain the indication information of the playback device in the semantic, and then determine the playback device corresponding to the semantic based on the indication information of the playback device in the semantic, and further send the semantics in the semantic set to the playback device corresponding to each semantic.

[0220] In another possible implementation, the server can obtain a second correspondence relationship, which includes the correspondence relationship between the timing information and the indication information of the playback device, and is used to indicate that the semantic containing the timing information can run in the playback device indicated by the indication information of the playback device. Therefore, the server can determine the playback device corresponding to each semantic according to the timing information of each semantic and the second correspondence relationship, and send the semantics to the corresponding playback device in the playback group in the order indicated by the timing information.

[0221] For example, the second correspondence relationship can be as shown in Table 5-1:

[0222] Table 5-1

[0223] Timing information Indication information of the playback device 1，3，5 HF-01 2，4，6 HF-02

[0224] In the example shown in Table 5-1, "HF-01" and "HF-02" represent the device identifier of the playback device; and Arabic numerals represent the timing information. Among them, the second row indicates that the semantics with timing information "1", "3" and "5" should be executed by the playback device with device identifier "HF-01"; and the third row indicates that the semantics with timing information "2", "4" and "6" should be executed by the playback device with device identifier "HF-02".

[0225] In addition, the server sends one semantic to the corresponding playback device at a time, or sends a group of semantics (including multiple semantics) to the corresponding playback device at a time when sending the semantics in the semantic set to the corresponding playback device. Optionally, the server can also receive a notification message from the playback device as a trigger condition for sending the next semantic (or the next group). The notification message indicates that the playback device has executed the received semantics, or it can be understood that the notification message indicates that the playback device has successfully played back the received semantics. For details, please refer to the related description in the foregoing step 207, which will not be described here.

[0226] In this embodiment, it is proposed to add timing information in the semantics for automation implementation as a basis for the server to schedule and distribute multiple semantics. After obtaining multiple semantics, the server can determine the order of distributing semantics according to the timing information of each semantic in the multiple semantics, and then distribute the multiple semantics to the corresponding playback device according to the order indicated by the timing information, so that each playback device can execute the semantics from the server in order, thereby realizing ordered playback. In this application, since the server can replace manual distribution of the multiple semantics in order, it can avoid confusion in the order of executing semantics by each playback device in the system, thereby ensuring that the automation process of the system including multiple devices can be realized.

[0227] In addition, as FIG. 5 indicated, the application also provides a server 50, FIG. 5 The structure of the server 50 provided in the application is shown in the figure. The server can be a cloud server or a local server. The server can provide services to users in the form of SaaS (Software as a Service). Users can access the services of the server 50 through web pages on recording devices, application software and plug-ins, open source codes or tool kits, etc. It should be understood that the foregoing FIG. 2 or FIG. 4 The server in the corresponding method embodiment can be based on the structure of the server 50 shown in the embodiment. FIG. 5

[0228] As FIG. 5 indicated, the server 50 can include a processor 510, a memory 520 and a transceiver 530. The processor 510 is coupled to the memory 520, and the processor 510 is coupled to the transceiver 530.

[0229] ​The transceiver 530 can also be referred to as a transceiving unit, a transceiver, a transceiving device, etc. Optionally, the device in the transceiving unit for implementing the receiving function can be regarded as a receiving unit, and the device in the transceiving unit for implementing the sending function can be regarded as a sending unit, that is, the transceiving unit includes the receiving unit and the sending unit, the receiving unit can also be referred to as a receiver, an input port, a receiving circuit, etc., and the sending unit can be referred to as a transmitter, a transmitter, or a transmitting circuit, etc. In this application, the transceiver 530 can receive a registration message (for example, a registration message from a recording device or a registration message from a playback device). The transceiver 530 can send a message to the recording device, for example, send a first recording instruction or a second recording instruction to the recording device, etc.

[0230] The processor 510 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 510 can refer to one processor, or can include multiple processors, which is not limited here.

[0231] In addition, the aforementioned memory 520 is mainly used for storing software programs and data. The memory 520 can exist independently and be connected to the processor 510. Alternatively, the memory 520 can be integrated with the processor 510, for example, in one or more chips. The memory 520 can store program codes for implementing the technical solutions of the embodiments of the present application and be controlled to execute by the processor 510. The executed computer programs of various types can also be regarded as the drivers of the processor 510. The memory 520 can include volatile memory (volatile memory), such as random access memory (random-access memory, RAM); the memory can also include non-volatile memory (non-volatile memory), such as read-only memory (read-only memory, ROM), flash memory (flash memory), hard disk drive (hard disk drive, HDD) or solid-state drive (solid-state drive, SSD); the memory 520 can also include a combination of the above types of memory. The memory 520 can refer to one memory or can include multiple memories. For example, the memory 520 is used to store the information of the recording device (for example, the device identifier and the device type of the recording device) and the information of the playback device (for example, the device identifier and the device type of the playback device). The memory 520 is also used to store the first correspondence or the second correspondence introduced above. The first correspondence is the correspondence between the indication information of each recording device in the recording group and the indication information of the playback device in the playback group, and the first correspondence is used to indicate that the playback device can execute the semantics generated by the corresponding recording device. The second correspondence includes the correspondence between the timing information and the indication information of the playback device, and the second correspondence is used to indicate that the semantics containing the timing information can run in the playback device indicated by the indication information of the playback device.

[0232] In an implementation manner, the memory 520 stores computer readable instructions, and the computer readable instructions include a plurality of software modules, such as a sending module 521, a processing module 522 and a receiving module 523. After the processor 510 executes each software module, the processor 510 can perform corresponding operations according to the instructions of each software module. In the present embodiment, the operation performed by one software module actually refers to the operation performed by the processor 510 according to the instructions of the software module.

[0233] Specifically, when the server 50 is used to execute the method in the corresponding embodiments FIG. 2 The main functions of the sending module 521, the processing module 522 and the receiving module 523 in the server 50 when the server 50 is used to execute the method in the corresponding embodiments are as follows:

[0234] The receiving module 523 is configured to receive at least one piece of semantics from each recording device in the recording group. The processing module 522 is configured to determine a time sequence of each piece of semantics according to time information of each piece of semantics, and control the transceiver module to send the semantics to a corresponding playback device in the playback group in the time sequence. Each piece of semantics is information describing an operation of a user on the recording device, and the semantics includes time information, object information, and action information. The time information is used to indicate a time at which the recording device detects the operation, the object information is used to indicate an object of the operation corresponding to the semantics, and the action information is used to indicate a content of the operation.

[0235] In a possible implementation, the memory 520 is configured to store a first correspondence relationship between indication information of each recording device in the recording group and indication information of a playback device in the playback group. The first correspondence relationship is used to indicate that the playback device can execute semantics generated by the corresponding recording device.

[0236] Optionally, the indication information includes a device identifier and / or a device type. The device type is used to indicate a software type and / or a hardware type supported by the recording device.

[0237] Optionally, the software type supported by the recording device and the software type supported by the corresponding playback device have a non-empty intersection, and / or the hardware type supported by the recording device and the hardware type supported by the corresponding playback device have a non-empty intersection.

[0238] In a possible implementation, when each piece of semantics includes indication information of a recording device, the processing module 522 is specifically configured to send the semantics to the corresponding playback device in the playback group in the time sequence according to the time information of each piece of semantics, the indication information of the recording device in each piece of semantics, and the first correspondence relationship.

[0239] In a possible implementation, the memory 520 in the server 50 is further configured to store a device list. The device list stores information of the recording group and information of the playback group. The information of the recording group includes a device identifier of each recording device in the recording group, and the information of the playback group includes a device identifier of each playback device in the playback group. At this time, the receiving module 523 is further configured to receive a plurality of registration messages. Each registration message includes a device identifier of a device sending the registration message. The processing module 522 is further configured to control the memory 520 to determine, according to the device list and the device identifier of the device in the registration message, whether the device is a recording device or a playback device.

[0240] In a possible implementation, the registration message further includes a device type of the device, the device type of the device including a device type of the recording device and a device type of the playback device. When the registration message further includes the device type of the device, the storage module in the server is further configured to store the device type of the recording device in correspondence with the device identifier of the recording device; and store the device type of the playback device in correspondence with the device identifier of the playback device.

[0241] In a possible implementation, the sending module 521 is further configured to send a first recording instruction to each of the recording devices in the recording group, the first recording instruction being used to instruct each of the recording devices to start recording, the first recording instruction including system clock information of the server, the system clock information being used to instruct each of the recording devices in the recording group to perform clock synchronization according to the system clock of the server.

[0242] In a possible implementation, the sending module 521 is further configured to send a second recording instruction to at least one of the recording devices in the recording group, the second recording instruction being used to instruct the recording device to stop recording.

[0243] Specifically, when the server 50 is configured to perform the method in the foregoing FIG. 4 The main functions of the sending module 521, the processing module 522, and the receiving module 523 in the server 50 when the server 50 is configured to perform the method in the corresponding embodiments are as follows: The processing module 522 controls the external interface to obtain a plurality of semantics. In addition, the processing module 522 controls the transceiver module to send each of the semantics to a corresponding playback device in the playback group according to an order indicated by time sequence information of each of the semantics. Each of the semantics is used to indicate an operation that needs to be performed on the playback device in the playback group, and the operation includes controlling one of the playback devices in the playback group to interact with another of the playback devices in the playback group. In addition, the semantics include time sequence information, object information, and action information. The time sequence information is used to indicate an order of the semantics relative to other semantics in the plurality of semantics, the object information is used to indicate an object of the operation corresponding to the semantics, and the action information is used to indicate a content of the operation.

[0244] In a possible implementation, the memory 520 in the server 50 stores indication information of each of the playback devices in the playback group, and the indication information of each of the playback devices is used to identify one of the playback devices in the playback group.

[0245] In a possible implementation, each of the semantics further includes indication information of the playback device, and the indication information of the playback device in the semantics is used to indicate the playback device corresponding to the semantics.

[0246] In a possible implementation, the processing module 522 is configured to control the transceiver module to send the semantics to the playback devices indicated by the indication information of the playback devices according to the time sequence information of each piece of the semantics and the indication information of the playback devices, in the order indicated by the time sequence information.

[0247] In a possible implementation, the processing module 522 controls the external interface to obtain a second correspondence relationship, where the second correspondence relationship includes a correspondence relationship between the time sequence information and the indication information of the playback devices, and the second correspondence relationship is used to indicate that the semantics containing the time sequence information can run on the playback devices indicated by the indication information of the playback devices. The processing module 522 is further configured to determine the corresponding playback devices of each piece of the semantics according to the time sequence information of each piece of the semantics and the second correspondence relationship, and control the transceiver module to send the semantics to the corresponding playback devices in the playback group in the order indicated by the time sequence information.

[0248] In a possible implementation, the indication information of the playback devices includes device identifiers of the playback devices and / or device types of the playback devices, where the device types of the playback devices are used to indicate software types and / or hardware types supported by the playback devices.

[0249] The remaining can be referred to FIG. 2 Or FIG. 4 The method of the server in the corresponding embodiment, which will not be repeated here.

[0250] For example, when the foregoing server 50 is applied to a multi-terminal automated scenario based on a user interface (UI), the processing module 522 in the server 50 can be divided into FIG. 6 structures shown. The processing module 522 includes a recording controller 5223, a semantic sorting and scheduling module 5222, and a playback controller 5221.

[0251] The recording controller 5223 is a functional module for controlling the recording process of the recording devices. Before recording, each recording device in the recording group registers through the recording controller 5223, and the server 50 receives the registration message through the receiving module 523. The registration message includes a device identifier of the recording device and a device type of the recording device. Optionally, the registration message further includes system clock information of the recording device. During the recording process, the recording controller 5223 issues an instruction (i.e., a first recording instruction) indicating to start recording and an instruction (i.e., a second recording instruction) indicating to stop recording, obtains semantics generated by each recording device in the recording group, and further obtains a semantic set containing a plurality of semantics.

[0252] The semantic sorting and scheduling module 5222 is the data storage and maintenance structure of server 50. It is used to sort the semantics in the semantic set according to time information, and then distribute them in the same chronological order during playback. Furthermore, the semantic sorting and scheduling module 5222 also stores a first correspondence, which is the correspondence between the indication information of each recording device in the recording group and the indication information of the playback devices in the playback group. This first correspondence is used to indicate that the playback device can execute the semantics generated by the corresponding recording device.

[0253] The playback controller 5221 is a functional module that controls the playback process. Before playback begins, each playback device in the playback group registers through the playback controller 5221. The server 50 receives the registration message through the receiving module 523. The registration message includes the device identifier and device type of the playback device. Optionally, the registration message also includes the system clock information of the playback device. During playback, the playback controller 5221 distributes the semantics in the aforementioned semantic set to each playback device in the playback group in chronological order. Optionally, the playback controller 5221 can also collect notification messages from the playback devices, indicating that the playback device has executed a certain semantic, so that the playback controller 5221 controls the sending module 521 to send the next semantic.

[0254] like FIG. 7 The diagram shown is a structural schematic of a device 70 provided in this application. The device 70 can be a recording device or a playback device. The aforementioned device can be a mobile terminal (MT) such as a mobile phone or tablet; it can also be a virtual reality (VR) terminal device, an augmented reality (AR) terminal device; or it can be a terminal device in other fields that may have automation needs, such as terminal devices in industrial control, self-driving, remote medical surgery, smart grids, transportation safety, smart cities, smart homes, etc. It should be understood that the aforementioned device 70 can also be other devices capable of executing scripts or semantics, or other devices with automation needs. FIG. 2 and FIG. 4The device (e.g., a recording device or a playback device) in the corresponding method embodiment can be based on FIG. 7 the structure of the recording device 70.

[0255] The recording device 70 includes at least one processor 701 and at least one memory 702. It should be understood that, FIG. 7 Only one processor 701 and one memory 702 are shown.

[0256] The processor 701 can be a general central processing unit CPU, a microprocessor, a network processor NP, or an application-specific integrated circuit, or one or more integrated circuits for controlling the execution of programs of the present application. The aforementioned processor 701 can be a single-CPU processor or a multi-CPU processor. The processor 701 can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). In addition, the processor 701 can be a separate semiconductor chip, or can be integrated with other circuits as a semiconductor chip, for example, it can be integrated with other circuits (such as coding and decoding circuits, hardware acceleration circuits, or various bus and interface circuits) to form a system-on-a-chip (SoC), or it can be integrated as a built-in processor of an application specific integrated circuit (ASIC), which can be packaged separately or packaged together with other circuits.

[0257] In addition, the aforementioned memory 702 can be a read-only memory ROM, or other types of static storage devices that can store static information and instructions, or a random access memory RAM, or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), which is not limited here. The memory 702 can exist independently, but is connected to the aforementioned processor 701. Alternatively, the memory 702 can be integrated with the aforementioned processor 701. For example, it can be integrated into one or more chips.

[0258] In addition, the memory 702 is also used to store program codes for implementing the technical solutions of the embodiments of the present application. For example, the mapping rule introduced in the foregoing embodiments, which includes a corresponding relationship between an action feature and a plurality of operation types, each of which satisfies the action feature.

[0259] In addition, the device 70 further includes a communication interface 703 for communicating with a server, so that the device can obtain the first recording instruction or the second recording instruction from the server. The device 70 can also send a registration message to the server through the communication interface 703.

[0260] The rest can be referred to FIG. 2 and FIG. 4 The method of the recording device and the playback device in the corresponding embodiments will not be described here.

[0261] When the device 70 is a recording device, the processor 701 can at least perform the functions of a monitor and an analyzer. The monitor is used to monitor the operation of the keyboard / key, mouse, touch screen and the like through the I / O interface to obtain raw information. Then, the raw information is transmitted to the analyzer. The analyzer generates semantics based on the raw information by searching and analyzing the mapping rule stored in the recording device. The specific way in which the recording device generates semantics can be referred to the related description in the foregoing step 205, which will not be described here.

[0262] When the device 70 is a playback device, the processor 701 can at least perform the functions of a parser and a UI driver. The parser is used to convert the semantics into instructions based on the received semantics and the mapping rule. Then, the parser transmits the instructions to the UI driver. The UI driver is used to reproduce the UI interaction situation based on the instructions.

[0263] It should be understood that, in the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the software form. The steps of the method disclosed in the embodiments of the present application can be directly embodied as the execution of the hardware processor, or executed by the combination of the hardware and software modules in the processor. The software module can be located in the random access memory, the flash memory, the read-only memory, the programmable read-only memory, the electrically erasable programmable memory, the register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here. It should also be understood that the first, second, third, fourth and various numerical numbers involved herein are only for the convenience of differentiation, and are not used to limit the scope of the embodiments of the present application.

[0264] Further, the present application provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are wholly or partially generated. For example, the server-related method in the foregoing FIG. 2 or FIG. 4 is implemented. For another example, the recording device-related method in the foregoing FIG. 2 is implemented. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium, or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be used by the computer or data storage device such as server, data center, etc. containing one or more available media sets. The available medium can be magnetic medium (for example, floppy disk, hard disk, magnetic tape), optical medium (for example, digital versatile disc (DVD)), or semiconductor medium (for example, solid state disk (SSD)) and the like.

[0265] Further, the present application also provides a computer readable storage medium storing a computer program, which is executed by a processor to implement the server-related method in the foregoing FIG. 2 or FIG. 4 .

[0266] Further, the present application also provides a computer readable storage medium storing a computer program, which is executed by a processor to implement the recording device-related method in the foregoing FIG. 2 .

[0267] It should be understood that the term "and / or" in the present application only describes the association relationship of the associated objects. The term "and / or" means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present application generally means that the front and rear associated objects have an "or" relationship.

[0268] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here again.

[0269] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A playback method characterized by, The method comprises: The server receives at least one semantic from each recording device in a recording group, each semantic being information describing an operation of a user on the recording device, the operation comprising controlling one recording device in the recording group to interact with another recording device in the recording group, the semantic comprising time information, object information and action information, the time information being used to indicate a time at which the recording device detects the operation, the object information being used to indicate an object of the operation corresponding to the semantic, and the action information being used to indicate a content of the operation; The server sends the semantic to a corresponding playback device in a playback group in a time sequence according to the time information of each semantic.

2. The method of claim 1, wherein, The server stores a first correspondence relationship between indication information of each recording device in the recording group and indication information of a playback device in the playback group, the first correspondence relationship being used to indicate that the playback device can execute a semantic generated by the corresponding recording device.

3. The method of claim 2, wherein, The indication information comprises a device identifier and / or a device type, the device type being used to indicate a software type and / or a hardware type supported by the recording device.

4. The method of claim 3, wherein, The software type supported by the recording device and the software type supported by the corresponding playback device have a non-empty intersection; and / or, the hardware type supported by the recording device and the hardware type supported by the corresponding playback device have a non-empty intersection.

5. The method according to any one of claims 2 to 4, characterized in that, Each semantic comprises the indication information of the recording device. The server sends the semantic to the corresponding playback device in the playback group in the time sequence according to the time information of each semantic, comprising: The server sends the semantic to the corresponding playback device in the playback group in the time sequence according to the time information of each semantic, the indication information of the recording device in each semantic and the first correspondence relationship.

6. The method according to any one of claims 2 to 5, characterized in that, The server stores a device list, the device list storing information of the recording group and information of the playback group, the information of the recording group comprising a device identifier of each recording device in the recording group, and the information of the playback group comprising a device identifier of each playback device in the playback group. Before the server receives at least one semantic from each recording device in the recording group, the method further comprises: The server receives a plurality of registration messages, each registration message comprising a device identifier of a device sending the registration message; The server determines, according to the device list and the device identifier of the device in the registration message, that the device is a recording device or a playback device.

7. The method of claim 6, wherein, The registration message further comprises a device type of the device, the device type of the device comprising a device type of the recording device and a device type of the playback device. The method further comprises: The server stores the device type of the recording device in correspondence with the device identifier of the recording device; The server stores the device type of the playback device in correspondence with the device identifier of the playback device.

8. The method according to claim 6 or 7, characterized in that, The method further comprises, before the server receives at least one semantic from each recording device in the recording group after the server receives the plurality of registration messages: The server sends a first recording instruction to each recording device in the recording group, the first recording instruction being used to instruct each recording device to start recording, the first recording instruction comprising system clock information of the server, the system clock information being used to instruct each recording device in the recording group to perform clock synchronization according to the system clock of the server.

9. The method of claim 8, wherein, The method further comprises, after the server sends the first recording instruction to each recording device in the recording group: The server sends a second recording instruction to at least one recording device in the recording group, the second recording instruction being used to instruct the recording device to stop recording.

10. A playback method characterized by, The method further comprises: The server obtains a plurality of semantics, each semantic being used to indicate an operation to be performed on a playback device in a playback group, the operation comprising controlling one playback device in the playback group to interact with another playback device in the playback group, the semantic comprising timing information, object information and action information, the timing information being used to indicate an order of the semantic relative to other semantics in the plurality of semantics, the object information being used to indicate an object of the operation corresponding to the semantic, and the action information being used to indicate a content of the operation. The server sends the semantic to a corresponding playback device in the playback group according to an order indicated by the timing information of each semantic.

11. The method of claim 10, wherein, The server stores indication information of each playback device in the playback group, the indication information of each playback device being used to identify one playback device in the playback group.

12. The method of claim 11, wherein, Each semantic further comprises indication information of the playback device, the indication information of the playback device in the semantic being used to indicate the playback device corresponding to the semantic.

13. The method of claim 12, wherein, The server sends the semantic to a corresponding playback device in the playback group according to an order indicated by the timing information of each semantic, comprising: The server sends the semantic to a playback device indicated by the indication information of the playback device according to the order indicated by the timing information of each semantic and the indication information of the playback device in each semantic.

14. The method of claim 11, wherein, The method further comprises, before the server sends the semantic to a corresponding playback device in the playback group according to an order indicated by the timing information of each semantic: The server obtains a second correspondence relationship, the second correspondence relationship comprising a correspondence relationship between the timing information and the indication information of the playback device, the second correspondence relationship being used to indicate that a semantic comprising the timing information can be run in a playback device indicated by the indication information of the playback device; The server sends the semantic to a corresponding playback device in the playback group according to an order indicated by the timing information of each semantic, comprising: The server sends the semantic to a corresponding playback device in the playback group according to an order indicated by the timing information of each semantic and the second correspondence relationship.

15. The method according to any one of claims 11 to 14, characterized in that, The indication information of the playback device includes a device identifier of the playback device and / or a device type of the playback device, and the device type of the playback device is used to indicate a software type and / or a hardware type supported by the playback device.

16. A recording method, characterized by, The method comprises: The recording device generates at least one semantic, each semantic being information generated by the recording device and describing an operation of a user on the recording device, the semantic including time information, object information, and action information, the time information being used to indicate a time at which the recording device detects the operation, the object information being used to indicate an object of the operation corresponding to the semantic, and the action information being used to indicate a content of the operation; The recording device sends the at least one semantic to a server, so that the server sends the semantic to a corresponding playback device in a playback group in chronological order according to the time information of each semantic.

17. The method of claim 16, wherein, The recording device is one of a plurality of recording devices in a recording group, and the operation includes controlling one recording device in the recording group to interact with another recording device in the recording group.

18. The method of claim 16 or 17, wherein, The recording device generates at least one semantic, comprising: The recording device obtains at least one original information, each original information being information detected by the recording device and related to an operation of a user on the recording device, each original information including operation time, operation object, and operation type; wherein the operation time is a time at which the recording device detects the operation of the user on the recording device, the operation object is an object detected by the recording device and operated by the user on the recording device, and the operation type is an action type detected by the recording device and performed by the user through hardware and / or software supported by the recording device. The recording device determines the operation time as the time information of the semantic. The recording device determines the object information of the semantic according to the operation object. The recording device determines the action information of the semantic according to the operation type.

19. The method of claim 18, wherein, The recording device determines the action information of the semantic according to the operation type, comprising: The recording device determines, according to the operation type and a mapping rule, that an action feature corresponding to the operation type is the action information of the semantic, the mapping rule including a corresponding relationship between one action feature and a plurality of operation types, and each operation type in the plurality of operation types satisfying the action feature.

20. The method of any one of claims 16-19, wherein, The semantic further includes a device type of the recording device.

21. The method of any one of claims 16 to 20, wherein, Before the recording device generates the at least one semantic, the method further comprises: The recording device sends a registration message to the server, the registration message including indication information of the recording device, and the indication information of the recording device including a device type of the recording device and a device identifier of the recording device.

22. The method of claim 21, wherein, After the recording device sends the registration message to the server, the method further comprises: The recording device receives a first recording instruction from the server, the first recording instruction being used to instruct the recording device to start recording, the first recording instruction comprising system clock information of the server, the system clock information being used to instruct each recording device in a recording group to perform clock synchronization according to the system clock of the server.

23. The method of claim 22, wherein, After the recording device receives the first recording instruction from the server, the method further comprises: The recording device receives a second recording instruction from the server, the second recording instruction being used to instruct the recording device to stop recording.

24. A server, comprising: comprising a processor and a memory; wherein the memory stores a computer program; The processor invokes the computer program to enable the server to perform the method according to any one of claims 1 to 9, or to perform the method according to any one of claims 10 to 15.

25. A recording device, characterized by comprising a processor and a memory; wherein the memory stores a computer program; The processor invokes the computer program to enable the recording device to perform the method according to any one of claims 16 to 23.

26. An automated system, comprising: comprising: a playback device, the server performing the method according to any one of claims 1 to 9, and the recording device performing the method according to any one of claims 16 to 23; a playback device, and the server performing the method according to any one of claims 10 to 15.

27. A computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 9, or to perform the method according to any one of claims 10 to 15, or to perform the method according to any one of claims 16 to 23.

28. A computer program product containing instructions that, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 9, or to perform the method according to any one of claims 10 to 15, or to perform the method according to any one of claims 16 to 23.

Images