Toy interaction action recognition system and method based on convolutional neural network

Abstract

The invention belongs to the field of intelligent toys, and discloses a toy interaction action recognition system and method based on a convolutional neural network, and the method comprises the steps: collecting an image of a user through video equipment, and obtaining the image frame number features of the image through a frame number extraction method; a touch sensor and a distance sensor which are arranged in the toy are used for obtaining touch features and distance features, and the image frame number features, the touch features, the distance features and corresponding toy interaction actions are used for constructing a toy neural network interaction model; the model processes real-time image slice features, real-time touch features and real-time distance features of a new user to obtain controlled new toy interaction actions, the new toy interaction actions are recognized and controlled according to the image, the contact condition and the distance of the user, and the practicability of the toy in the aspect of interaction action recognition is improved; according to the toy under the control method, the user condition can be accurately recognized, corresponding response is made, and therefore the user experience of the toy is improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent toy interaction, and in particular relates to a toy interaction action recognition system and method based on convolutional neural network. Background Art

[0002] Toy interactive action recognition mainly uses human-computer interaction. Human-computer interaction refers to the study of the interactive relationship between the system and the user to improve the system's usability and user-friendliness. The system broadly refers to various machines and equipment or computerized systems and software. Users can interact with the system in a friendly manner through the human-computer interaction interface. Usually a display screen is installed on the robot, and users can interact with the robot through the display screen to exchange information.

[0003] Existing improvements in interactive recognition of toys usually involve improvements in voice recognition in interactive recognition. For example, in a Chinese patent with application publication number CN113920993A, a method and device for intelligent awakening of a voice interactive plush toy are disclosed. This solution provides an intelligent awakening method for a voice interactive plush toy, which makes the voice interactive plush toy proactive and improves the user experience. Other improvements in interactive recognition of toys usually involve directly inputting recorded images into conventional machine learning algorithms such as convolutional neural networks, and only performing interactive actions with gestures. This will result in the toy being unable to accurately capture the user's action information when interacting with the user and thus unable to respond in time, thereby affecting the user experience. In view of this, it is necessary to improve the existing toy interactive action recognition technology.

[0004] Through the above analysis, the existing technology has the following problems to be solved:

[0005] How to utilize the video image information of users using toys at any time and process the image data to make fuller use of the image features to identify and control the toy interaction actions, how to more perfectly use multi-category data types to control the toy interaction actions, and how to use corresponding improved models for specific toy interaction scenarios to accurately identify and control the toy interaction actions are issues that need to be urgently addressed in the field of smart toy interaction.

[0006] Therefore, it is necessary to design a toy interactive action recognition method and system based on convolutional neural networks to address existing problems. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a toy interactive action recognition system and method based on convolutional neural network.

[0008] In a first aspect of the present invention, a method for identifying toy interactive actions based on a convolutional neural network is proposed, which is characterized by:

[0009] Acquire a first user image according to the video device, acquire a first tactile feature according to the tactile sensor device, acquire a first distance feature according to the distance sensor device, and acquire a first toy interaction action;

[0010] Processing the first user image to obtain a first image frame number feature, and processing it with the first tactile feature and the first distance feature to obtain a first toy interaction feature;

[0011] constructing a toy neural network interaction model using the first toy interaction feature and the corresponding first toy interaction action, wherein the toy neural network interaction model is a deep learning model;

[0012] A second user image acquired by a video device is received, the second user image is processed to obtain a second image frame number feature, a smart toy hand with an embedded tactile sensor and a distance sensor collects a second tactile feature and a second distance feature, processes them to obtain a second toy interaction feature, a toy neural network interaction model is called to process the second toy interaction feature to obtain a second toy interaction action, and the toy action recognition and control is performed according to the second toy interaction action.

[0013] Furthermore, the first user image is processed to obtain a first image frame number feature or the second user image is processed to obtain a second image frame number feature using the following method: the first image frame number feature or the second image frame number feature is obtained by obtaining the grayscale value of the x column in a Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image at the i-second corresponding to the f-frame picture, and the grayscale value of the y row in a Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image at the i-second corresponding to the f-frame picture.

[0014] Further, the first toy interaction feature is determined by the pressure adjustment coefficient, the temperature adjustment coefficient and the frame adjustment coefficient determined by the first image frame feature, the first tactile feature, the first distance feature and the gender of the toy user;

[0015] The second toy interaction feature is determined by a pressure adjustment coefficient, a temperature adjustment coefficient and a frame adjustment coefficient determined by the second image frame feature, the second tactile feature, the second distance feature and the gender of the toy user.

[0016] A toy interactive action recognition system based on a convolutional neural network is also provided, characterized in that the toy interactive action recognition system includes a toy interactive video acquisition module, a tactile sensor module, a distance sensor module, an intelligent toy interactive feature processing module, a toy neural network interactive model module and a toy action recognition control module:

[0017] The toy interaction video acquisition module is a video monitoring device that collects the first user image or the second user image in the toy placement area;

[0018] The tactile sensor module: acquires the first tactile feature or the second tactile feature through the tactile sensor device;

[0019] The distance sensor module: obtains the first distance feature or the second distance feature through the distance sensor device;

[0020] The intelligent toy interaction feature processing module processes the first user image to obtain a first image frame feature, and processes it with the first tactile feature and the first distance feature to obtain a first toy interaction feature, and also processes the second user image to obtain a second image frame feature, and processes it with the second tactile feature and the second distance feature to obtain a second toy interaction feature;

[0021] The toy action recognition control module is used to obtain a first toy interaction action and a second toy interaction action; and perform toy action recognition control according to the first toy interaction action or the second toy interaction action;

[0022] The toy neural network interaction model module is connected to the toy action recognition control module, receives the first toy interaction action, and uses the first toy interaction feature and the corresponding first toy interaction action to build a toy neural network interaction model, wherein the toy neural network interaction model is a deep learning model; and is also used to receive the second toy interaction feature in the intelligent toy interaction feature processing module, and the toy neural network interaction model processes the second toy interaction feature to obtain the second toy interaction action, and the second toy interaction action is transmitted to the toy action recognition control module.

[0023] Furthermore, the first user image is processed to obtain a first image frame number feature or the second user image is processed to obtain a second image frame number feature using the following method: the first image frame number feature or the second image frame number feature is obtained by obtaining the grayscale value of the x column in a Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image at the i-second corresponding to the f-frame picture, and the grayscale value of the y row in a Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image at the i-second corresponding to the f-frame picture.

[0024] Further, the first toy interaction feature is determined by the pressure adjustment coefficient, the temperature adjustment coefficient and the frame adjustment coefficient determined by the first image frame feature, the first tactile feature, the first distance feature and the gender of the toy user;

[0025] The second toy interaction feature is determined by a pressure adjustment coefficient, a temperature adjustment coefficient and a frame adjustment coefficient determined by the second image frame feature, the second tactile feature, the second distance feature and the gender of the toy user.

[0026] The present invention uses video surveillance equipment to obtain video image information of authorized users using toys at any time, and uses a frame number extraction method to process image data so that image features can be more fully and accurately used to identify and control toy interactive actions. The toy's built-in tactile sensors and distance sensors are more comprehensively used to collect multiple categories of data types combined with image feature data to control toy interactive actions, and the activation function of the neural network model is improved for specific toy interactive scenarios, so that the nonlinear expression ability of the model is more suitable for this scenario. The improved model used can enhance the accurate identification and control of toy interactive actions. The toy interactive actions are identified and controlled according to the user's image, contact situation and distance, thereby improving the practicality of the toy in identifying interactive actions. The toy under the control method of the invention can accurately identify the user's situation and respond accordingly, thereby improving the user experience of the toy.

[0027] More embodiments and improved effects of the present invention will be further introduced in conjunction with the drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of a toy interactive action recognition method based on a convolutional neural network of the present invention;

[0029] Figure 2 It is a schematic diagram of a toy interactive action recognition system based on a convolutional neural network of the present invention;

[0030] Figure 3 It is a schematic diagram of a Cartesian coordinate system feature representation of a frame of an image in an embodiment of the present invention;

[0031] Figure 4 Schematic diagram of activation function of CNN model before improvement in the embodiment of the present invention;

[0032] Figure 5 It is a schematic diagram of the structure of an electronic device for implementing the method of the present invention according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The invention is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0034] In order to solve the above technical problems, the present invention proposes a toy interactive action recognition system and method based on convolutional neural network.

[0035] In a first aspect of the present invention, a method for identifying toy interactive actions based on a convolutional neural network is proposed, which is characterized by:

[0036] Acquire a first user image according to the video device, acquire a first tactile feature according to the tactile sensor device, acquire a first distance feature according to the distance sensor device, and acquire a first toy interaction action;

[0037] Processing the first user image to obtain a first image frame number feature, and processing it with the first tactile feature and the first distance feature to obtain a first toy interaction feature;

[0038] constructing a toy neural network interaction model using the first toy interaction feature and the corresponding first toy interaction action, wherein the toy neural network interaction model is a deep learning model;

[0039] A second user image acquired by a video device is received, the second user image is processed to obtain a second image frame number feature, a smart toy hand with an embedded tactile sensor and a distance sensor collects a second tactile feature and a second distance feature, processes them to obtain a second toy interaction feature, a toy neural network interaction model is called to process the second toy interaction feature to obtain a second toy interaction action, and the toy action recognition and control is performed according to the second toy interaction action.

[0040] Furthermore, the first image frame number feature or the second image frame number feature is obtained by the grayscale value of the x column in the Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image corresponding to the f frame picture at the ith second, and the image grayscale value of the y row in the Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image corresponding to the f frame picture at the ith second.

[0041] In this embodiment, the first user image is processed to obtain the first image frame number feature or the second user image is processed to obtain the second image frame number feature using the following method. The specific calculation formula of the first image frame number feature or the second image frame number feature S is:

[0042]

[0043] Where G x represents the gray value of the first user image or the second user image corresponding to the f-th frame image at the ith second, in the Cartesian coordinate system constructed with the lower left corner of the image as the origin, with the horizontal coordinate being the x column, y represents the image grayscale value of the first user image or the second user image at the ith second corresponding to the fth frame of the image, with the Cartesian coordinate system constructed with the lower left corner of the image as the origin and the horizontal coordinate being the yth row, n is the maximum value of the image on the x-axis, m is the maximum value of the image on the y-axis, and M if |(Gx , G t )| is G x and G t The maximum value of , K is the frame number adjustment coefficient.

[0044] In this embodiment, if the user video is 10 minutes long, the value of i is 1-600, if the number of frames is 24, the value of f is 1-24, and the frame adjustment coefficient K is 2.

[0045] Furthermore, a method for calculating the first tactile feature or the second tactile feature T is as follows:

[0046] T=n1P+n2R

[0047] Wherein, n1 and n2 are the pressure adjustment coefficient and temperature adjustment coefficient determined according to the gender of the toy user, P is the pressure value collected by the tactile sensor, and R is the perceived temperature collected by the tactile sensor.

[0048] In this embodiment, since the effort and body temperature of men and women are different, an adjustment coefficient is used for correction, and the values ​​of n1 are 0.2 and 0.4, and the values ​​of n2 are 0.6 and 0.8. If it is a female, n1 is 0.2 and n2 is 0.8, and if it is a male, n1 is 0.4 and n2 is 0.6.

[0049] Further, the first toy interaction feature is determined by the pressure adjustment coefficient, the temperature adjustment coefficient and the frame adjustment coefficient determined by the first image frame feature, the first tactile feature, the first distance feature and the gender of the toy user;

[0050] The second toy interaction feature is determined by a pressure adjustment coefficient, a temperature adjustment coefficient and a frame adjustment coefficient determined by the second image frame feature, the second tactile feature, the second distance feature and the gender of the toy user.

[0051] In this example, the method of processing the first image frame number feature, the first tactile feature and the first distance feature to obtain the first toy interaction feature or the method of processing the second image frame number feature, the second tactile feature and the second distance feature to obtain the second toy interaction feature Z is as follows:

[0052]

[0053] In the formula, S is the first image frame number feature or the second image frame number feature, T is the first tactile feature or the second tactile feature, L is the first distance feature or the second distance feature, n1 and n2 are the pressure adjustment coefficient and the temperature adjustment coefficient determined according to the gender of the toy user, and K is the frame number adjustment coefficient.

[0054] Furthermore, the toy neural network interaction model is constructed using an improved convolutional neural network model, and its activation function is:

[0055]

[0056] f(Z) is the activation function value, S is the first image frame number feature or the second image frame number feature, T is the first tactile feature or the second tactile feature, L is the first distance feature or the second distance feature, and Z is the first toy interaction feature or the second toy interaction feature.

[0057] A toy interactive action recognition system based on a convolutional neural network is also provided, characterized in that the toy interactive action recognition system includes a toy interactive video acquisition module, a tactile sensor module, a distance sensor module, an intelligent toy interactive feature processing module, a toy neural network interactive model module and a toy action recognition control module:

[0058] The toy interaction video acquisition module is a video monitoring device that collects the first user image or the second user image in the toy placement area;

[0059] The tactile sensor module: acquires the first tactile feature or the second tactile feature through the tactile sensor device;

[0060] The distance sensor module: obtains the first distance feature or the second distance feature through the distance sensor device;

[0061] The intelligent toy interaction feature processing module processes the first user image to obtain a first image frame feature, and processes it with the first tactile feature and the first distance feature to obtain a first toy interaction feature, and also processes the second user image to obtain a second image frame feature, and processes it with the second tactile feature and the second distance feature to obtain a second toy interaction feature;

[0062] The toy action recognition control module is used to obtain a first toy interaction action and a second toy interaction action; and perform toy action recognition control according to the first toy interaction action or the second toy interaction action;

[0063] The toy neural network interaction model module is connected to the toy action recognition control module, receives the first toy interaction action, and uses the first toy interaction feature and the corresponding first toy interaction action to build a toy neural network interaction model, wherein the toy neural network interaction model is a deep learning model; and is also used to receive the second toy interaction feature in the intelligent toy interaction feature processing module, and the toy neural network interaction model processes the second toy interaction feature to obtain the second toy interaction action, and the second toy interaction action is transmitted to the toy action recognition control module.

[0064] Furthermore, the first image frame number feature or the second image frame number feature is obtained by the grayscale value of the x column in the Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image corresponding to the f frame picture at the ith second, and the image grayscale value of the y row in the Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image corresponding to the f frame picture at the ith second.

[0065] In this embodiment, the first user image is processed to obtain the first image frame number feature or the second user image is processed to obtain the second image frame number feature using the following method. The specific calculation formula of the first image frame number feature or the second image frame number feature S is:

[0066]

[0067] Where G x represents the gray value of the first user image or the second user image corresponding to the f-th frame image at the ith second, in the Cartesian coordinate system constructed with the lower left corner of the image as the origin, with the horizontal coordinate being the x column, y represents the image grayscale value of the first user image or the second user image at the ith second corresponding to the fth frame of the image, with the Cartesian coordinate system constructed with the lower left corner of the image as the origin and the horizontal coordinate being the yth row, n is the maximum value of the image on the x-axis, m is the maximum value of the image on the y-axis, and M if |(G x , G t )| is G x and G t The maximum value of , K is the frame number adjustment coefficient.

[0068] Furthermore, the calculation method of the first tactile feature or the second tactile feature T is as follows:

[0069] T=n1P+n2R

[0070] Wherein, n1 and n2 are the pressure adjustment coefficient and temperature adjustment coefficient determined according to the gender of the toy user, P is the pressure value collected by the tactile sensor, and R is the perceived temperature collected by the tactile sensor.

[0071] Further, the first toy interaction feature is determined by the pressure adjustment coefficient, the temperature adjustment coefficient and the frame adjustment coefficient determined by the first image frame feature, the first tactile feature, the first distance feature and the gender of the toy user;

[0072] The second toy interaction feature is determined by a pressure adjustment coefficient, a temperature adjustment coefficient and a frame adjustment coefficient determined by the second image frame feature, the second tactile feature, the second distance feature and the gender of the toy user.

[0073] In this embodiment, the method of processing the first image frame number feature, the first tactile feature and the first distance feature to obtain the first toy interaction feature or the method of processing the second image frame number feature, the second tactile feature and the second distance feature to obtain the second toy interaction feature Z is as follows:

[0074]

[0075] In the formula, S is the first image frame number feature or the second image frame number feature, T is the first tactile feature or the second tactile feature, L is the first distance feature or the second distance feature, n1 and n2 are the pressure adjustment coefficient and the temperature adjustment coefficient determined according to the gender of the toy user, and K is the frame number adjustment coefficient.

[0076] Furthermore, the toy neural network interaction model is constructed using an improved convolutional neural network model, and its activation function is:

[0077]

[0078] f(Z) is the activation function value, S is the first image frame number feature or the second image frame number feature, T is the first tactile feature or the second tactile feature, L is the first distance feature or the second distance feature, and Z is the first toy interaction feature or the second toy interaction feature.

[0079] The present invention uses video surveillance equipment to obtain video image information of authorized users using toys at any time, and uses a frame number extraction method to process image data so that image features can be more fully and accurately used to identify and control toy interactive actions. The toy's built-in tactile sensors and distance sensors are more comprehensively used to collect multiple categories of data types combined with image feature data to control toy interactive actions, and the activation function of the neural network model is improved for specific toy interactive scenarios, so that the nonlinear expression ability of the model is more suitable for this scenario. The improved model used can enhance the accurate identification and control of toy interactive actions. The toy interactive actions are identified and controlled according to the user's image, contact situation and distance, thereby improving the practicality of the toy in identifying interactive actions. The toy under the control method of the invention can accurately identify the user's situation and respond accordingly, thereby improving the user experience of the toy.

[0080] The present invention does not particularly specify the structure of some modules, which shall be subject to the contents recorded in the prior art. The prior art mentioned in the above background technology section and the specific embodiment section of the present invention can be used as part of the present invention to understand the meaning of some technical features or parameters. The scope of protection of the present invention shall be subject to the contents actually recorded in the claims.

Claims

1. A toy interactive action recognition method based on convolutional neural network, characterized in that: Acquire a first user image according to the video device, acquire a first tactile feature according to the tactile sensor device, acquire a first distance feature according to the distance sensor device, and acquire a first toy interaction action; Processing the first user image to obtain a first image frame number feature, and processing it with the first tactile feature and the first distance feature to obtain a first toy interaction feature; constructing a toy neural network interaction model using the first toy interaction feature and the corresponding first toy interaction action, wherein the toy neural network interaction model is a deep learning model; A second user image acquired by a video device is received, the second user image is processed to obtain a second image frame number feature, a smart toy hand with an embedded tactile sensor and a distance sensor collects a second tactile feature and a second distance feature, processes them to obtain a second toy interaction feature, a toy neural network interaction model is called to process the second toy interaction feature to obtain a second toy interaction action, and the toy action recognition and control is performed according to the second toy interaction action.

2. A toy interactive action recognition method based on convolutional neural network as claimed in claim 1, characterized in that: The first user image is processed to obtain a first image frame number feature or the second user image is processed to obtain a second image frame number feature using the following method: the first image frame number feature or the second image frame number feature is obtained by the grayscale value of the x column in the Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image corresponding to the f frame picture at the ith second, and the image grayscale value of the y row in the Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image corresponding to the f frame picture at the ith second.

3. A toy interactive action recognition method based on convolutional neural network as claimed in claim 2, characterized in that: The first toy interaction feature is determined by a pressure adjustment coefficient, a temperature adjustment coefficient and a frame adjustment coefficient determined by the first image frame feature, the first tactile feature, the first distance feature and the gender of the toy user.

4. A toy interactive action recognition method based on convolutional neural network as claimed in claim 3, characterized in that: The second toy interaction feature is determined by a pressure adjustment coefficient, a temperature adjustment coefficient and a frame adjustment coefficient determined by the second image frame feature, the second tactile feature, the second distance feature and the gender of the toy user.

5. A toy interactive action recognition system based on convolutional neural network, characterized in that: The toy interactive action recognition system includes a toy interactive video acquisition module, a tactile sensor module, a distance sensor module, an intelligent toy interactive feature processing module, a toy neural network interactive model module, and a toy action recognition control module: The toy interaction video acquisition module is a video monitoring device that collects the first user image or the second user image in the toy placement area; The tactile sensor module: acquires the first tactile feature or the second tactile feature through the tactile sensor device; The distance sensor module: obtains the first distance feature or the second distance feature through the distance sensor device; The intelligent toy interaction feature processing module processes the first user image to obtain a first image frame feature, and processes it with the first tactile feature and the first distance feature to obtain a first toy interaction feature, and also processes the second user image to obtain a second image frame feature, and processes it with the second tactile feature and the second distance feature to obtain a second toy interaction feature; The toy action recognition control module is used to obtain a first toy interaction action and a second toy interaction action; and perform toy action recognition control according to the first toy interaction action or the second toy interaction action; The toy neural network interaction model module is connected to the toy action recognition control module, receives the first toy interaction action, and uses the first toy interaction feature and the corresponding first toy interaction action to build a toy neural network interaction model, wherein the toy neural network interaction model is a deep learning model; and is also used to receive the second toy interaction feature in the intelligent toy interaction feature processing module, and the toy neural network interaction model processes the second toy interaction feature to obtain the second toy interaction action, and the second toy interaction action is transmitted to the toy action recognition control module.

6. A toy interactive action recognition system based on convolutional neural network as claimed in claim 5, characterized in that: The first user image is processed to obtain a first image frame number feature or the second user image is processed to obtain a second image frame number feature using the following method: the first image frame number feature or the second image frame number feature is obtained by the grayscale value of the x column in the Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image corresponding to the f frame picture at the ith second, and the image grayscale value of the y row in the Cartesian coordinate system constructed with the lower left corner of the picture as the origin for the first user image or the second user image corresponding to the f frame picture at the ith second.

7. A toy interactive action recognition system based on convolutional neural network as claimed in claim 6, characterized in that: The first toy interaction feature is determined by a pressure adjustment coefficient, a temperature adjustment coefficient and a frame adjustment coefficient determined by the first image frame feature, the first tactile feature, the first distance feature and the gender of the toy user.

8. A toy interactive action recognition system based on convolutional neural network as claimed in claim 7, characterized in that: The second toy interaction feature is determined by a pressure adjustment coefficient, a temperature adjustment coefficient and a frame adjustment coefficient determined by the second image frame feature, the second tactile feature, the second distance feature and the gender of the toy user.

Images