Control instruction generation method and apparatus

Abstract

The application discloses a control instruction generation method and device, and relates to the technical field of intelligent interaction. A specific embodiment of the method comprises: in response to determining that a specified image contains a registered object, acquiring an image frame sequence corresponding to the specified image; determining a target registered object and a target wrist point corresponding thereto based on the detection times of each registered object in the image frame sequence; tracking the motion trajectory of the target wrist point and the target registered object, and generating a control instruction for a smart electronic device based on the motion trajectory. This embodiment helps to improve the convenience, flexibility and effectiveness of interaction.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the field of intelligent interaction technology, and in particular to a method and apparatus for generating control instructions. Background Technology

[0002] With the development of intelligent television terminals, people have an increasing demand for natural and efficient human-computer interaction methods. Currently, the main technologies that can achieve human-computer interaction in the field include voice recognition, gesture recognition, human motion recognition, and the use of customized hardware devices.

[0003] However, existing human-computer interaction technologies suffer from problems such as high cost, limited device substitutability, over-reliance on customized hardware, and low recognition accuracy. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for generating control instructions.

[0005] According to a first aspect, embodiments of this application provide a control instruction generation method, the method comprising: in response to determining that a specified image contains a registered object, acquiring an image frame sequence corresponding to the specified image; determining a target registered object and its corresponding target wrist point based on the number of detections of the registered object in the image frame sequence; tracking the motion trajectory of the target wrist point and the target registered object, and generating a control instruction for a smart electronic device based on the motion trajectory.

[0006] According to a second aspect, embodiments of this application provide a control instruction generation apparatus, including: an acquisition module configured to acquire an image frame sequence corresponding to the specified image in response to determining that a specified image contains a registered object; and a determination module configured to determine a target registered object and a corresponding target wrist point based on the number of times the registered object is detected in the image frame sequence.

[0007] The generation module is configured to track the motion trajectory of the target wrist point and the target registered object, and generate control commands for the smart electronic device based on the motion trajectory.

[0008] According to a third aspect, embodiments of this application provide an electronic device including one or more processors; a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a control instruction generation method as described in any embodiment of the first aspect.

[0009] According to a fourth aspect, embodiments of this application provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements a control instruction generation method as described in any embodiment of the first aspect.

[0010] This application, in response to determining that a specified image contains a registered object, acquires an image frame sequence corresponding to the specified image; based on the detection count of each registered object in the image frame sequence, determines the target registered object and its corresponding target wrist point; tracks the motion trajectory of the target wrist point and the target registered object, and generates control commands for the intelligent electronic device based on the motion trajectory, thereby improving the convenience, flexibility and effectiveness of the interaction.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0013] Figure 2 This is a flowchart of an embodiment of the control instruction generation method according to this application;

[0014] Figure 3a , 3b This is a schematic diagram of an application scenario of the control instruction generation method according to this application;

[0015] Figure 4 This is a schematic diagram of another application scenario of the control instruction generation method according to this application;

[0016] Figure 5 This is a schematic diagram of another application scenario of the control instruction generation method according to this application;

[0017] Figure 6 This is a schematic diagram of another application scenario of the control instruction generation method according to this application;

[0018] Figure 7 This is a flowchart of yet another embodiment of the control instruction generation method according to this application;

[0019] Figure 8 This is a schematic diagram of an embodiment of the control instruction generation apparatus according to this application;

[0020] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the server embodiments of this application.

[0021] The specific implementation methods are as follows:

[0022] Exemplary embodiments of this application are described in conjunction with the accompanying drawings, including various details of the embodiments to aid understanding; however, these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Figure 1 An exemplary system architecture 100 is shown, in which an embodiment of the control instruction generation method of this application can be applied.

[0025] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0026] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Client application software, such as video playback applications and communication applications, can be installed on terminal devices 101, 102, and 103.

[0027] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, smart bracelets, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0028] Server 105 can be a server that provides various services, such as: in response to determining that a specified image contains a registered object, acquiring an image frame sequence corresponding to the specified image; determining the target registered object and its corresponding target wrist point based on the number of detections of each registered object in the image frame sequence; tracking the motion trajectory of the target wrist point and the target registered object, and generating control instructions for the intelligent electronic device based on the motion trajectory.

[0029] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide control command generation services), or as a single software program or software module. No specific limitations are made here.

[0030] It should be noted that the control command generation method provided in the embodiments of this disclosure can be executed by server 105, terminal devices 101, 102, and 103, or by server 105 and terminal devices 101, 102, and 103 in cooperation with each other. Accordingly, all parts (e.g., units, subunits, modules, and submodules) of the control command generation device can be entirely located in server 105, entirely located in terminal devices 101, 102, and 103, or separately located in server 105 and terminal devices 101, 102, and 103.

[0031] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0032] Figure 2 A flowchart 200 illustrates an embodiment of the control command generation method applicable to this application. The control command generation method includes the following steps:

[0033] Step 201: In response to determining that the specified image contains a registered object, obtain the image frame sequence corresponding to the specified image.

[0034] In this embodiment, the execution entity (e.g., Figure 1 The server 105 or terminal devices 101, 102, 103 can acquire specified images in real time or periodically via the image acquisition device. After acquiring the specified image, the executing entity can directly match the objects contained in the specified image with the data in the registered object database to determine whether the specified image contains a registered object; or it can first determine the candidate region of the specified image, match the objects in the candidate region of the specified image with the data in the registered object database to determine whether the objects in the candidate region of the specified image are registered objects. If the objects in the specified image are registered objects, then the image frame sequence corresponding to the specified image is further acquired.

[0035] The number of registered objects contained in the specified image can be one or more, and this application does not limit this.

[0036] The aforementioned registered object database is used to record and store data such as registered objects, their characteristics, corresponding priority levels, and control commands. Users register objects that meet the requirements according to preset rules, and select the corresponding priority, control commands, etc., which are then stored in the registered object database. The successfully registered objects are called registered objects.

[0037] Here, preset rules can be set based on experience and actual needs, such as preset shaped objects (e.g., regular polyhedra, spherical bodies), preset objects with specified features on their surfaces, etc.

[0038] In addition, preset rules may also include object priority, corresponding control commands, etc., which are not limited in this application.

[0039] The designated image is used to indicate the image of the environment within a preset range acquired by the image acquisition device from the intelligent electronic device.

[0040] Here, the image acquisition device can be any device that converts optical information into electrical signals for storage, whether it is a device in the existing technology or a future technology, such as a camera or video camera, and this application does not limit it.

[0041] The execution subject can determine the candidate region of the specified image by directly determining the entire specified image as the candidate region, by determining the candidate region based on the wrist point in the specified image, or by first determining the candidate region based on the wrist point, and if there is no wrist point, by determining the entire specified image as the candidate region. This application does not limit this.

[0042] In some alternative methods, in response to determining that a specified image contains a registered object, the image frame sequence corresponding to the specified image is obtained: in response to determining that a specified image contains a registered object based on a registered object database, the image frame sequence corresponding to the specified image is obtained.

[0043] In this implementation, the executing entity can match the objects contained in the specified image with the data in the registered object database to determine whether the specified image contains a registered object; if a matching registered object exists in the registered object database, it is determined that the specified image contains a registered object, and the image frame sequence corresponding to the specified image is further obtained.

[0044] The registered object database can be generated in the following way: obtain images of each face of the object to be registered according to the preset rules; extract features from the image of each face of the object to be registered to obtain feature vectors; insert the feature vectors into the database to obtain the registered object database.

[0045] Here, preset rules include objects with preset shapes. For example, the preset rule is a regular polyhedron (such as a cuboid, cube, etc.), and the registered object database can include Rubik's Cubes, item boxes, etc. registered by the user according to the preset rules.

[0046] Specifically, the smart electronic device is a television. The executing entity can acquire a specified image of the environment within a preset range of the smart television through a camera set on the smart television. The feature vector of the object (such as a Rubik's Cube) in the specified image is matched with the feature vector of the registered object (Rubik's Cube, item box, etc., which correspond to objects with preset rules of regular polyhedra) in the registered object database. If a matching feature vector exists, the object in the specified image is determined to be a registered object, that is, the Rubik's Cube in the specified image is determined to be a registered object, and the image frame sequence corresponding to the specified image is further acquired.

[0047] Here, for each face of the object to be registered, the executing entity can first determine the region of interest in the image, and then input the region of interest into a preset neural network to extract features.

[0048] The pre-defined neural network can be trained using a first target loss function, a second target loss function, and training samples. The first target loss function can include a first loss function, a second loss function, and a third loss function. The first loss function is constructed based on the positional features of the object; the second loss function is constructed based on the type of pre-defined rules corresponding to the object (the types of pre-defined rules can include multiple types, such as regular polyhedra (e.g., cubes, cuboids, etc.), spheres, etc.); and the third loss function is constructed based on whether the appearance of the object conforms to the target pre-defined rules (e.g., cuboids, cubes).

[0049] The first objective loss function L1 can be expressed by the following formula:

[0050]

[0051] The second objective loss function can be constructed based on the feature vectors corresponding to the training samples, the preset positive sample feature vectors corresponding to the training samples, and the preset negative sample feature vectors corresponding to the training samples.

[0052] The second objective loss function L2 can be expressed by the following formula:

[0053]

[0054] Where N indicates the total number of samples. Indicates the feature vector corresponding to the training sample. Indicates the predefined positive sample feature vector corresponding to the training sample. Indicates the pre-defined negative sample feature vector corresponding to the training sample. These are preset parameters.

[0055] Here, the second objective loss function helps the neural network get closer to positive samples and further away from negative samples, thereby improving the accuracy of the trained neural network.

[0056] This implementation method determines whether a specified image contains a registered object based on a registered object database. The registered object database can be generated by: acquiring images of each face of the object to be registered according to preset rules; extracting features from the image of each face of the object to be registered to obtain a feature vector; and inserting the feature vector into the database to obtain the registered object database. The preset rules include objects of preset shapes, which effectively improves the efficiency and accuracy of registered object recognition.

[0057] Step 202: Based on the number of detections of each registered object in the image frame sequence, determine the target registered object and its corresponding target wrist point.

[0058] In this embodiment, after obtaining the image frame sequence, the executing entity can count the number of detections of each registered object in the image frame sequence. For example, it can count the number of detections of each registered object in the corresponding candidate region in the image frame sequence, and determine the target registered object based on the detection data. For example, it can determine the registered object with the most detections as the target registered object, and determine the wrist point corresponding to the target registered object as the target wrist point.

[0059] Specifically, the image frame sequence includes 15 image frames, and the candidate region of each image frame is the entire image. Among them, the number of detections of registered object A is 13, the number of detections of registered object B is 2, and the number of detections of registered object A is the highest. Therefore, registered object A can be identified as the target registered object, and the wrist point corresponding to registered object A can be identified as the target wrist point.

[0060] It should be noted that the target registered object can be one registered object or multiple registered objects, and this application does not limit this.

[0061] Step 203: Track the motion trajectory of the target wrist point and the target registered object, and generate control commands for the smart electronic device based on the motion trajectory.

[0062] In this embodiment, after determining the target wrist point and the target registered object, the executing entity can track the movement trajectory of the target wrist point and the target registered object, and generate control commands for the intelligent electronic device based on the movement trajectory.

[0063] Here, the executing entity can determine the control command for the intelligent electronic device based on the motion trajectory and the preset motion trajectory and control command comparison table; or it can input the motion trajectory into the preset command generation model to obtain the control command, which is not limited in this application.

[0064] The instruction generation model can be trained based on motion trajectory samples labeled with control instructions.

[0065] Here, the trajectory of motion can be a stationary motion, or it can be a single action. For example, for a single registered target object, it can be a movement in eight linear directions (up, down, left, right, upper left, upper right, lower left, lower right), or a movement in two rotational directions (rotating to the left and rotating to the right). For two registered target objects, it can be a linear relative movement of moving closer to or further away from each other, or a relative movement of two registered target objects turning left or right simultaneously. It can also be a series of actions, such as drawing a specific shape, like a square or a star. This application does not limit this.

[0066] Specifically, for example, in a TV volume control scenario, an upward movement increases the volume, while a downward movement decreases it. In a TV program control scenario, a leftward rotation indicates returning to the previous screen; a rightward movement indicates moving to the selected item; and a rightward rotation indicates entering the selected item. In a photo album control scenario, a leftward movement indicates flipping through the album to the left, while a rightward movement indicates flipping through the album to the right. In a game control scenario, an upward or downward movement indicates controlling the target object to jump.

[0067] Furthermore, it should be noted that there may be one or more control commands, and this application does not limit this.

[0068] In some alternative approaches, control commands for the intelligent electronic device are generated based on the motion trajectory, including: extracting a first feature of the motion trajectory of the target wrist point and a second feature of the motion trajectory of the target registered object; fusing the first feature and the second feature to obtain a fused feature; and generating control commands for the intelligent electronic device based on the fused feature.

[0069] In this implementation, the executing entity can extract the first feature of the motion trajectory of the target wrist point and the second feature of the motion trajectory of the target registered object; fuse the first feature and the second feature according to their respective weights to obtain the fused feature; and determine the control command for the intelligent electronic device based on the fused feature and a preset lookup table of fused features and control commands.

[0070] Specifically, the trajectory of the target object is as follows: Figure 3a As shown, the trajectory of the target wrist point is as follows: Figure 3b As shown.

[0071] The trajectory of a target object mainly includes: 1. Movement distance: dist obj2. Travel time: time obj 3. Angle of movement: angle obj .

[0072] In addition, it also includes direction:

[0073] Predefined direction:

[0074] Directional weights:

[0075] The first feature can be obtained through Characterization

[0076]

[0077] in, According to time obj and angle obj Sure, and Used to indicate a preset feature threshold.

[0078] The movement trajectory of the target wrist point mainly includes: 1. Movement distance: 2. Travel time: 3. Angle of movement: 4. Aspect Ratio: .

[0079] In addition, it also includes direction:

[0080] Predefined direction:

[0081] Key point stability weights:

[0082] Static time interval weight:

[0083] Directional weights:

[0084] Second feature Characterization

[0085]

[0086]

[0087]

[0088]

[0089] in, According to and Sure, , , , , Used to indicate a preset feature threshold.

[0090] Here, the weights corresponding to the first feature and the second feature are respectively The fusion feature D can be represented by the following formula.

[0091]

[0092] This implementation method extracts the first feature of the motion trajectory of the target wrist point and the second feature of the motion trajectory of the target registered object, respectively; it then fuses the first and second features to obtain a fused feature, and generates control commands for the intelligent electronic device based on the fused feature. This fully considers the different influences of the first and second features on the fused feature, which helps to improve the accuracy of the determined control commands.

[0093] In some alternative approaches, the target registered object and its corresponding target wrist point are determined based on the number of times the registered object is detected in the image frame sequence, including: determining the target registered object and its corresponding target wrist point based on the number of times each registered object is detected in the image frame sequence and the priority of the registered object.

[0094] In this implementation, after obtaining the image frame sequence, the executing entity can count the number of detections of each registered object in the image frame sequence, and further determine the target registered object and its corresponding target wrist point based on the detection data and corresponding weights, the priority of the registered object and its corresponding weights.

[0095] In addition, the executing entity can also count the number of times registered objects are detected in the candidate region corresponding to the image frame sequence, and further determine the target candidate region based on the detection data and corresponding weights, the priority of the registered objects and their corresponding weights, and determine the registered objects contained in the target candidate region and their corresponding wrist points as the target registered objects and their corresponding target wrist points.

[0096] This implementation determines the target registered object and its corresponding target wrist point based on the number of detections of each registered object in the image frame sequence and the priority of the registered object. It fully considers the priority of the registered object, which helps to improve the accuracy of the determined target registered object and its corresponding target wrist point.

[0097] In some optional methods, in response to determining that a specified image contains a registered object, obtaining an image frame sequence corresponding to the specified image includes: in response to determining that a candidate region of the specified image contains a registered object, obtaining an image frame sequence corresponding to the specified image.

[0098] In this implementation, the candidate regions of a specified image are determined based on wrist points within the specified image. After acquiring the specified image, the executing entity can use a deep learning model from existing or future technologies to detect wrist points in the specified image, determine candidate regions based on the detected wrist points, and further match objects in the candidate regions of the specified image with a preset registered object database to determine whether an object in the candidate region of the specified image is a registered object. If an object in the specified image is a registered object, the corresponding image frame sequence of the specified image is further acquired.

[0099] This implementation narrows the detection range of registered objects by determining candidate regions based on wrist points in a specified image, thereby improving detection efficiency and ultimately increasing the efficiency of generating control commands for intelligent electronic devices.

[0100] In some alternative approaches, control commands for the intelligent electronic device are generated based on the motion trajectory, including: in response to determining that the motion trajectory is a preset action sequence, generating control commands for the intelligent electronic device based on the preset action sequence.

[0101] In this implementation, after determining the target wrist point and the target registered object, the executing entity can track the movement trajectory of the target wrist point and the target registered object and identify the movement trajectory. If the movement trajectory is a preset action sequence, that is, a preset action feature sequence, then control instructions for the intelligent electronic device can be generated according to the preset action sequence and control instruction lookup table.

[0102] The preset action sequence can be set according to experience and actual needs, such as the action sequence for drawing a star, the action sequence for drawing a square, etc. This application does not limit this.

[0103] Specifically, such as Figure 4 As shown, users can also customize action sequences 403 and 404 based on the predetermined action movement rule database 401 and the appearance database 402 of each registered object. For example, an action sequence for drawing a star, an action sequence for drawing a circle, etc. The executing entity identifies the motion trajectory. If the identified motion trajectory is a preset action sequence, such as the action sequence for drawing a star, then a control command for the intelligent electronic device 405 to enter game 1 is generated according to the preset action sequence. If the action sequence for drawing a circle is generated, then a control command for the intelligent electronic device 406 to enter game 2 is generated according to the preset action sequence.

[0104] This implementation generates control commands for smart electronic devices by responding to a predetermined motion trajectory as a preset action sequence, which helps to generate control commands based on customized action sequences.

[0105] In some alternative approaches, control commands for the intelligent electronic device are generated based on motion trajectories, including: generating control commands for the intelligent electronic device based on the motion trajectories of two target registered objects and their corresponding target wrist points.

[0106] In this implementation, there are two target registered objects. After determining the target wrist point and the target registered object, the executing entity can track the trajectory of the two target registered objects and the corresponding target wrist point, and generate control commands for the intelligent electronic device according to the preset motion trajectory and control command comparison table.

[0107] Specifically, such as Figure 5 As shown, in the racing game control scenario, the executing entity can control the racing car 503 to turn left 504, turn right 505, and go straight 506 in the game according to the movement trajectory of two target registered objects 501 and 502 and their corresponding target wrist points.

[0108] This implementation generates control commands for smart electronic devices by registering two target objects and their corresponding target wrist points, which helps to improve the flexibility and diversity of the generated control commands.

[0109] See also Figure 6 , Figure 6 This is a schematic diagram of an application scenario of the control instruction generation method according to this embodiment.

[0110] exist Figure 6 In the application scenario, the execution entity 601 can acquire a specified image within a preset range of the smart electronic device 602, such as a television, via an image acquisition device 603, such as a camera, and match the objects contained in the specified image with a preset registered object database. In response to determining that the acquired specified image contains a registered object, the entity acquires the image frame sequence corresponding to the specified image. Based on the number of detections of each registered object in the image frame sequence, the entity determines the target registered object, such as a cube, and the corresponding target wrist point. The entity tracks the motion trajectory of the target wrist point and the target registered object, and based on the motion trajectory, generates control commands for the smart electronic device, such as a command to increase the volume of the television program.

[0111] The control command generation method provided in the embodiments of this disclosure obtains an image frame sequence corresponding to the specified image in response to determining that a specified image contains a registered object; determines the target registered object and its corresponding target wrist point based on the detection count of each registered object in the image frame sequence; tracks the motion trajectory of the target wrist point and the target registered object; and generates control commands for the intelligent electronic device based on the motion trajectory. This realizes the generation of control commands based on the user's operation on a preset object with a regular shape, without the need to identify what the object is, effectively improving the convenience, flexibility and effectiveness of the interaction.

[0112] Further reference Figure 7 This illustrates a flow 700 of another embodiment of the control command generation method. In this embodiment, the flow 700 of the control command generation method may include the following steps:

[0113] Step 701: In response to determining that the specified image contains a registered object, obtain the image frame sequence corresponding to the specified image.

[0114] In this embodiment, the implementation details and technical effects of step 701 can be found in the description of step 201, and will not be repeated here.

[0115] Step 702: Based on the number of detections of each registered object in the image frame sequence, determine the target registered object and its corresponding target wrist point.

[0116] In this embodiment, the implementation details and technical effects of step 702 can be found in the description of step 202, and will not be repeated here.

[0117] Step 703: Based on the motion trajectory and at least one display surface presented by the target registered object during the motion, generate control instructions for the intelligent electronic device.

[0118] In this embodiment, after determining the target wrist point and the target registered object, the executing entity can track the movement trajectory of the target wrist point and the target registered object, and generate control commands for the intelligent electronic device based on the movement trajectory and one or more display surfaces presented by the target registered object during the movement.

[0119] Here, the executing entity can determine the control command for the intelligent electronic device based on the motion trajectory, the display surface of the target registered object during the motion, and a preset comparison table of motion trajectory, display surface of the registered object, and control command; alternatively, the motion trajectory and the display surface of the target registered object during the motion can be input into a preset command generation model to obtain the control command. This application does not limit this.

[0120] The instruction generation model can be trained based on target samples labeled with control instructions. The target samples include the motion trajectory of the registered object and at least one display surface of the registered object during the motion.

[0121] Specifically, for example, the target registered object is a cube with six faces: A, B, C, D, E, and F. Each face has a different pattern. The trajectory of the target registered object and its corresponding target wrist point is upward movement. During the movement, if the displayed face of the target registered object is A, the corresponding control command is to launch application 1; if the displayed face is B, the corresponding control command is to launch application 2; if the displayed face is C, the corresponding control command is to launch application 3; if the displayed face is D, the corresponding control command is to launch application 4; if the displayed face is E, the corresponding control command is to launch application 5; and if the displayed face is F, the corresponding general control command is used.

[0122] For example, the target registered object is a cube with six faces, each with a different pattern. The movement trajectory of the target registered object and its corresponding target wrist point is to remain stationary for a preset time. During the movement, if the displayed face of the target registered object is one of the six faces, the game mode corresponding to that face is activated and the game begins. Here, the game modes corresponding to the six faces are all different.

[0123] In some alternative approaches, control instructions for the intelligent electronic device are generated based on the motion trajectory and at least one display surface presented by the target registered object during the motion, including: generating control instructions for the intelligent electronic device based on the motion trajectory, at least two display surfaces presented by the target registered object during the motion, and the presentation order of the at least two display surfaces.

[0124] In this implementation, after determining the target wrist point and the target registered object, the executing entity can track the movement trajectory of the target wrist point and the target registered object, and generate control instructions for the intelligent electronic device based on the movement trajectory, at least two display surfaces presented by the target registered object during the movement, and the presentation order of the at least two display surfaces.

[0125] Specifically, the target registration object is a cube with six faces: A, B, C, D, E, and F. Each face has a different pattern. The movement trajectory of the target registration object and its corresponding target wrist point is tracked, and the displayed face of the target registration object during the movement is recorded. If the displayed face of the target registration object is A or B, the corresponding control command is to launch application 6; if the displayed face of the target registration object is C or B, the corresponding control command is to save the program.

[0126] This implementation method, based on the motion trajectory, the at least two display surfaces of the target registered object during the motion process, and the presentation order of the at least two display surfaces, helps to achieve customized and diversified command control of smart electronic devices.

[0127] The above embodiments of this application, and Figure 2 Compared to the corresponding embodiments, the process 700 of the control command generation method in this embodiment reflects the generation of control commands for intelligent electronic devices based on the motion trajectory and at least one display surface presented by the target registered object during the motion. This helps users to experience the shape attributes of objects in the virtual environment and further enhances the flexibility and diversity of interaction.

[0128] Further reference Figure 8 As an implementation of the methods shown in the above figures, this application provides an embodiment of a control command generation device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0129] like Figure 8 As shown, the control command generation device 800 of this embodiment includes: an acquisition module 801, a determination module 802, and a generation module 803.

[0130] The acquisition module 801 can be configured to acquire the image frame sequence corresponding to the specified image in response to determining that the specified image contains a registered object.

[0131] The determination module 802 can be configured to determine the target registered object and its corresponding target wrist point based on the number of times each registered object is detected in the image frame sequence.

[0132] The generation module 803 can be configured to track the motion trajectory of the target wrist point and the target registered object, and generate control commands for the smart electronic device based on the motion trajectory.

[0133] In some optional embodiments of this example, the generation module is further configured to generate control commands for the smart electronic device based on the motion trajectory and at least one display surface of the target registered object during the motion.

[0134] In some optional embodiments of this example, the generation module is further configured to generate control commands for the smart electronic device based on the motion trajectory, at least two display surfaces of the target registered object during the motion, and the presentation order of the at least two display surfaces.

[0135] In some optional embodiments of this example, the generation module is further configured to: extract a first feature of the motion trajectory of the target wrist point and a second feature of the motion trajectory of the target registered object; fuse the first feature and the second feature to obtain a fused feature; and generate control instructions for the smart electronic device based on the fused feature.

[0136] In some optional embodiments of this example, the acquisition module is further configured to: in response to determining that a candidate region of the acquired specified image contains a registered object, acquire the image frame sequence corresponding to the specified image.

[0137] In some optional embodiments of this example, the determining module is further configured to: determine the target registered object and its corresponding target wrist point based on the number of detections of each registered object in the image frame sequence and the priority of the registered object.

[0138] In some optional embodiments of this example, the generation module is further configured to: in response to determining that the motion trajectory is a preset action sequence, generate control instructions for the intelligent electronic device based on the preset action sequence.

[0139] In some optional embodiments of this example, the generation module is further configured to generate control commands for the smart electronic device based on the motion trajectories of the two target registered objects and their corresponding target wrist points.

[0140] In some optional embodiments of this example, the acquisition module is further configured to: in response to determining that a specified image contains a registered object based on the registered object database, acquire the image frame sequence corresponding to the specified image.

[0141] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0142] like Figure 9 The diagram shown is a block diagram of an electronic device according to the control instruction generation method of an embodiment of this application.

[0143] 900 is a block diagram of an electronic device according to a control instruction generation method according to embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0144] like Figure 9 As shown, the electronic device includes one or more processors 901, a memory 902, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 9 Take the 901 processor as an example.

[0145] The memory 902 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the control instruction generation method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the control instruction generation method provided in this application.

[0146] Memory 902, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the control instruction generation method in the embodiments of this application (e.g., appendix). Figure 8 The acquisition module 801, determination module 802, and generation module 803 are shown. The processor 901 executes various functional applications and data processing of the server by running non-transient software programs, instructions, and modules stored in the memory 902, thereby implementing the control instruction generation method in the above method embodiment.

[0147] Memory 902 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device generated by control instructions. Furthermore, memory 902 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 902 may optionally include memory remotely located relative to processor 901, and these remote memories can be connected to the electronic device that generates control instructions via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0148] The electronic device for the control instruction generation method may further include an input device 903 and an output device 904. The processor 901, memory 902, input device 903, and output device 904 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0149] Input device 903 can receive input digital or character information, as well as key signal input related to user settings and function control of electronic devices for quality monitoring of live video streams, such as touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output device 904 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0150] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0153] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0154] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0155] The technical solutions according to the embodiments of this application help to improve the convenience, flexibility and effectiveness of interaction.

[0156] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0157] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for generating control instructions, the method comprising: In response to determining that the candidate region of the acquired specified image contains a registered object pre-stored in the registered object database, the image frame sequence corresponding to the specified image is acquired. The specified image is used to indicate the environment within a preset range of the smart electronic device. The registered object is an object with a regular shape. The registered object database is constructed based on images of multiple different objects to be registered according to preset rules. The candidate region of the specified image is determined based on the wrist point in the specified image. Based on the number of detections of each registered object in the image frame sequence, the target registered object and its corresponding target wrist point are determined. The movement trajectory of the target wrist point and the target registered object is tracked, and control commands for the intelligent electronic device are generated based on the movement trajectory.

2. The method according to claim 1, wherein, The step of generating control commands for the intelligent electronic device based on the motion trajectory includes: Based on the motion trajectory and at least one display surface of the target registered object during the motion, control commands for the intelligent electronic device are generated.

3. The method according to claim 2, wherein, The step of generating control commands for the smart electronic device based on the motion trajectory and at least one display surface of the target registered object during the motion includes: Based on the motion trajectory, at least two display surfaces of the target registered object during the motion, and the presentation order of the at least two display surfaces, control instructions for the intelligent electronic device are generated.

4. The method according to claim 1, wherein, The step of generating control commands for the intelligent electronic device based on the motion trajectory includes: Extract the first feature of the motion trajectory of the target wrist point and the second feature of the motion trajectory of the target registered object, respectively. The first feature and the second feature are fused to obtain the fused feature; Based on the fusion features, control commands for intelligent electronic devices are generated.

5. The method according to claim 1, wherein, The step of determining the target registered object and its corresponding target wrist point based on the number of detections of each registered object in the image frame sequence includes: Based on the number of detections of each registered object in the image frame sequence and the priority of each registered object, the target registered object and its corresponding target wrist point are determined.

6. The method according to claim 1, wherein, The step of generating control commands for the intelligent electronic device based on the motion trajectory includes: In response to determining that the motion trajectory is a preset action sequence, control commands for the intelligent electronic device are generated based on the preset action sequence.

7. The method according to claim 1, wherein, The number of target registered objects is two, and the generation of control commands for the intelligent electronic device based on the motion trajectory includes: Based on the motion trajectories of the two target registered objects and their corresponding target wrist points, control commands for the intelligent electronic device are generated.

8. The method according to claim 1, wherein, The step of obtaining the image frame sequence corresponding to the specified image in response to determining that the candidate region of the acquired specified image contains registered objects pre-stored in the registered object database includes: In response to determining that a specified image contains a registered object based on a registered object database, an image frame sequence corresponding to the specified image is obtained. The registered object database is generated in the following manner: images of each face of the object to be registered according to a preset rule are obtained; features are extracted from the image of each face of the object to be registered to obtain a feature vector; the feature vector is inserted into the database to obtain the registered object database. The preset rule includes objects of preset shapes.

9. A control command generation apparatus, the apparatus comprising: The acquisition module is configured to acquire an image frame sequence corresponding to a specified image in response to determining that the candidate region of the acquired specified image contains a registered object pre-stored in a registered object database. The specified image is used to indicate the environment within a preset range of the smart electronic device. The registered object is an object with a regular shape. The registered object database is constructed based on images of multiple different objects to be registered according to a preset rule. The candidate region of the specified image is determined based on a wrist point in the specified image. The determination module is configured to determine the target registered object and its corresponding target wrist point based on the number of times each registered object is detected in the image frame sequence. The generation module is configured to track the motion trajectory of the target wrist point and the target registered object, and generate control commands for the smart electronic device based on the motion trajectory.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores information that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images