Robotic arm control method, device and electronic equipment based on sound information

Abstract

The present application provides a method, device and electronic device for controlling a robotic arm based on sound information, and relates to the field of robot control technology. In the present application, target sound information is obtained from the collected environmental sound information of the environment in which the robotic arm is located; the target sound information includes motion control keywords for controlling the movement of the robotic arm; feature extraction is performed on the target sound information to obtain a first feature set and a second feature set; wherein the first feature set includes multiple physical features of the target sound information, and the second feature set includes multiple semantic features of the target sound information; based on the first feature set and the second feature set, the target performance data of the robotic arm and the action instruction set associated with the target performance data are determined; based on the action instruction set, the robotic arm is controlled to display the target performance data. It can be seen that real-time adjustment and dynamic response of the robotic arm's movements can be achieved according to the target sound information, thereby increasing the interactivity and flexibility in the robotic arm control process.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a robot arm control method, device and electronic equipment based on sound information. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology and robotics technology, robotic arms, as a common basic form of robots, have been widely used in many fields of daily life. In relevant application scenarios, robotic arms usually complete a series of actions corresponding to a pre-set action instruction according to the action instruction.

[0003] Therefore, when adopting the above-mentioned robot arm control method, since the pre-set action instructions are fixed control instructions, this will cause the series of actions corresponding to the action instructions completed by the robot arm to be fixed, and the interactivity is poor, making it difficult to meet the robot arm control needs of different users.

[0004] In view of this, how to achieve good interaction between the robotic arm and sound information during the robotic arm control process is a problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a method, device and electronic device for controlling a robotic arm based on sound information, so as to achieve good interaction between the robotic arm and the sound information during the control process of the robotic arm.

[0006] In a first aspect, an embodiment of the present application provides a method for controlling a robotic arm based on sound information, the method comprising:

[0007] Obtaining target sound information from the collected environmental sound information of the environment in which the robot arm is located; the target sound information includes motion control keywords for controlling the motion of the robot arm;

[0008] Extracting features of the target sound information to obtain a first feature set and a second feature set; wherein the first feature set includes a plurality of physical features of the target sound information, and the second feature set includes a plurality of semantic features of the target sound information;

[0009] Determining target performance data of the robot arm and a set of motion instructions associated with the target performance data based on the first feature set and the second feature set;

[0010] Based on the action instruction set, the robotic arm is controlled to display the target performance data.

[0011] In an optional embodiment, obtaining target sound information from the collected ambient sound information of the environment in which the robot arm is located includes:

[0012] Performing noise reduction processing on the ambient sound information, filtering out the ambient noise included in the ambient sound information, and obtaining the ambient sound information after the noise reduction processing;

[0013] The ambient sound information after noise reduction is normalized to obtain the target sound information.

[0014] In an optional embodiment, feature extraction is performed on the target sound information to obtain a first feature set and a second feature set, including:

[0015] Based on the sound feature extraction methods respectively set for the multiple physical features, extract the sound features of the target sound information to obtain a first feature set; and

[0016] Perform semantic analysis on the target sound information to obtain a second feature set.

[0017] In an optional embodiment, the first feature set includes any one or a combination of pitch features, volume features, timbre features, speaking speed features, duration features and spatial position features; wherein the duration features are used to characterize the duration of the target sound information, and the spatial position features are used to characterize the sound source position coordinates corresponding to the target sound information.

[0018] In an optional embodiment, determining the target performance data of the robot arm and the action instruction set associated with the target performance data based on the first feature set and the second feature set includes:

[0019] Determining, from a preset candidate performance data set, target performance data that matches a target semantic feature representing a motion control keyword in a second feature set;

[0020] Determine an action parameter subset corresponding to each action in the target performance data based on a mapping relationship between preset features and action parameters, and a plurality of physical features included in the first feature set and at least one semantic feature in the second feature set except the target semantic feature;

[0021] An action instruction set is generated based on the obtained multiple action parameter subsets.

[0022] In an optional embodiment, generating an action instruction set based on the obtained multiple action parameter subsets includes:

[0023] For multiple action parameter subsets, perform the following operations respectively:

[0024] Based on the conversion relationship between the preset action parameters and the action instructions, generate a first action instruction corresponding to the first action parameter subset; wherein the first action parameter subset is any one of the multiple action parameter subsets;

[0025] The first action instruction is saved in the action instruction set.

[0026] In an optional embodiment, controlling the mechanical arm to display target performance data based on the action instruction set includes:

[0027] Controlling the robot arm to display multiple actions included in the target performance data according to the instruction execution order of multiple action instructions included in the action instruction set;

[0028] After each action of the robot arm, the following operations are performed:

[0029] Obtain feedback information of the action performed by the robot arm at the current moment; the feedback information is used to indicate the completion status of the action, as well as the first feature subset and the second feature subset of the next moment adjacent to the current moment;

[0030] Modifying the mapping relationship between the preset features and the action parameters and the conversion relationship between the preset action parameters and the action instructions based on the feedback information to obtain a modified mapping relationship and a modified conversion relationship;

[0031] The motion instruction at the next moment is adjusted based on the modified mapping relationship and the modified conversion relationship.

[0032] In an optional embodiment, modifying the mapping relationship between the preset features and the action parameters and the conversion relationship between the preset action parameters and the action instructions based on the feedback information includes:

[0033] Determine a first modification factor of the mapping relationship and a second modification factor of the conversion relationship based on the feedback information;

[0034] The mapping relationship is modified based on the first modification factor, and the conversion relationship is modified based on the second modification factor.

[0035] In a second aspect, an embodiment of the present application further provides a robotic arm control device based on sound information, the device comprising:

[0036] An information acquisition module is used to acquire target sound information from the collected environmental sound information of the environment in which the robot arm is located; the target sound information includes motion control keywords for controlling the motion of the robot arm;

[0037] A feature extraction module, used to extract features from the target sound information to obtain a first feature set and a second feature set; wherein the first feature set includes a plurality of physical features of the target sound information, and the second feature set includes a plurality of semantic features of the target sound information;

[0038] An instruction determination module, used to determine target performance data of the robot arm and an action instruction set associated with the target performance data based on the first feature set and the second feature set;

[0039] The command control module is used to control the robot arm to display target performance data based on the action command set.

[0040] In an optional embodiment, when acquiring the target sound information from the collected ambient sound information of the environment in which the robot arm is located, the information acquisition module is specifically used to:

[0041] Performing noise reduction processing on the ambient sound information, filtering out the ambient noise included in the ambient sound information, and obtaining the ambient sound information after the noise reduction processing;

[0042] The ambient sound information after noise reduction is normalized to obtain the target sound information.

[0043] In an optional embodiment, when extracting features from the target sound information to obtain the first feature set and the second feature set, the feature extraction module is specifically used to:

[0044] Based on the sound feature extraction methods respectively set for the multiple physical features, extract the sound features of the target sound information to obtain a first feature set; and

[0045] Perform semantic analysis on the target sound information to obtain a second feature set.

[0046] In an optional embodiment, when determining the target performance data of the robot arm and the action instruction set associated with the target performance data based on the first feature set and the second feature set, the instruction determination module is specifically used to:

[0047] Determining, from a preset candidate performance data set, target performance data that matches a target semantic feature representing a motion control keyword in a second feature set;

[0048] Determine an action parameter subset corresponding to each action in the target performance data based on a mapping relationship between preset features and action parameters, and a plurality of physical features included in the first feature set and at least one semantic feature in the second feature set except the target semantic feature;

[0049] An action instruction set is generated based on the obtained multiple action parameter subsets.

[0050] In an optional embodiment, when generating the action instruction set based on the obtained multiple action parameter subsets, the instruction determination module is specifically used to:

[0051] For multiple action parameter subsets, perform the following operations respectively:

[0052] Based on the conversion relationship between the preset action parameters and the action instructions, generate a first action instruction corresponding to the first action parameter subset; wherein the first action parameter subset is any one of the multiple action parameter subsets;

[0053] The first action instruction is saved in the action instruction set.

[0054] In an optional embodiment, when controlling the mechanical arm to display target performance data based on the action instruction set, the instruction control module is specifically used to:

[0055] Controlling the robot arm to display multiple actions included in the target performance data according to the instruction execution order of multiple action instructions included in the action instruction set;

[0056] After each action of the robot arm, the following operations are performed:

[0057] Obtain feedback information of the action performed by the robot arm at the current moment; the feedback information is used to indicate the completion status of the action, as well as the first feature subset and the second feature subset of the next moment adjacent to the current moment;

[0058] Modifying the mapping relationship between the preset features and the action parameters and the conversion relationship between the preset action parameters and the action instructions based on the feedback information to obtain a modified mapping relationship and a modified conversion relationship;

[0059] The motion instruction at the next moment is adjusted based on the modified mapping relationship and the modified conversion relationship.

[0060] In an optional embodiment, when modifying the mapping relationship between the preset features and the action parameters and the conversion relationship between the preset action parameters and the action instructions based on the feedback information, the instruction control module is specifically used to:

[0061] Determine a first modification factor of the mapping relationship and a second modification factor of the conversion relationship based on the feedback information;

[0062] The mapping relationship is modified based on the first modification factor, and the conversion relationship is modified based on the second modification factor.

[0063] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0064] Processor; and

[0065] Memory for storing programs,

[0066] The program includes instructions, which, when executed by a processor, cause the processor to execute the robot arm control method based on sound information as described in the first aspect.

[0067] In a fourth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the robotic arm control method based on sound information as described in the first aspect.

[0068] In a fifth aspect, the present application provides a computer program product, which, when called by a computer, enables the computer to execute the steps of the robot arm control method based on sound information as described in the first aspect.

[0069] The beneficial effects of this application are as follows:

[0070] In the robot arm control method based on sound information provided in the embodiment of the present application, after obtaining the collected environmental sound information of the environment in which the robot arm is located, the target sound information included in the environmental sound information can be feature extracted to obtain a first feature set including multiple physical features and a second feature set including multiple semantic features, thereby determining the target performance data of the robot arm and the action instruction set associated with the target performance data based on the first feature set and the second feature set, and then controlling the robot arm to display the target performance data based on the action instruction set. It can be seen that the real-time adjustment and dynamic response of the robot arm action can be achieved based on the target sound information, thereby increasing the interactivity and flexibility in the robot arm control process.

[0071] In addition, other features and advantages of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described here are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0073] Figure 1 A schematic diagram of an implementation flow of a robotic arm control method based on sound information applicable to an embodiment of the present application;

[0074] Figure 2 A logical schematic diagram of feature extraction of target sound information provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of an implementation flow of a method for determining target performance data of a robotic arm and its corresponding action instruction set provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of the structure of a robotic arm control device based on sound information provided in an embodiment of the present application;

[0077] Figure 5A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0079] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0080] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0081] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0082] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0083] First, the design concept of the embodiment of the present application is briefly introduced below:

[0084] With the continuous development of AI technology and robotics technology, robots are gradually moving from traditional industrial applications to many areas closer to life, such as services and entertainment. Among them, robot performances (such as robot dance) as a novel form of performance have attracted more and more attention. In related technologies, the implementation of robot dance mainly relies on pre-compiled program instructions to control the robot to complete a series of fixed actions. Although this method can realize robot dance, it lacks interactivity and cannot adjust the dance movements in real time according to changes in the external environment, making it difficult to bring a better viewing experience to the audience.

[0085] In view of this, how to achieve good interaction between the robot arm and sound information during the control process of the robot arm, which is a common basic form of the robot, that is, to improve interactivity, is a problem that needs to be solved urgently. Therefore, in order to improve the interactivity in the process of controlling the robot arm, the embodiment of the present application proposes a robot arm control method based on sound information, which may specifically include: first, obtaining target sound information from the collected environmental sound information of the environment in which the robot arm is located; the target sound information includes motion control keywords that control the movement of the robot arm; then, feature extraction is performed on the target sound information to obtain a first feature set and a second feature set; wherein the first feature set may include multiple physical features of the target sound information, and the second feature set may include multiple semantic features of the target sound information; further, based on the first feature set and the second feature set, the target performance data of the robot arm and the action instruction set associated with the target performance data are determined; finally, the robot arm is controlled to display the target performance data based on the action instruction set.

[0086] It should be noted that the execution subject of the sound information-based robotic arm control method provided in the embodiment of the present application may be one or more communication devices, which is not limited in the embodiment of the present application. Among them, the communication device may be a terminal device or server capable of controlling the robotic arm, and of course it may also be other communication devices, which is not limited in the embodiment of the present application.

[0087] Terminal device is a device that can provide voice and / or data connectivity to users, and can be a device that supports wired and / or wireless connection. Exemplarily, terminal devices can include but are not limited to: mobile phones, tablet computers, laptops, PDAs, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in unmanned driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc. In addition, a related client can be installed on the terminal device, and the client can be software, such as an application (APP), a browser, a short video software, etc., or a web page, a small program, etc.

[0088] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0089] The following describes a robot arm control method based on sound information provided by an exemplary embodiment of the present application in conjunction with the accompanying drawings. Figure 1 As shown, it is a schematic diagram of an implementation process of a robotic arm control method based on sound information provided in an embodiment of the present application. The execution subject takes a server as an example. The specific implementation process of the method is as follows:

[0090] S101: Obtain target sound information from the collected ambient sound information of the environment in which the robot arm is located.

[0091] The target sound information may include motion control keywords for controlling the movement of the robotic arm. For example, the target sound information is "Spring is here", where "Spring" may be a motion control keyword for controlling the movement of the robotic arm. After obtaining the motion control keywords included in the target sound information, the server may determine the performance data to be subsequently displayed by the robotic arm, for example, the robotic arm will display a dance program or a drama interface related to "Spring".

[0092] For example, when executing step S101, the server can collect the environmental sound information of the environment in which the robot is located through the sound collection module on the terminal device, thereby receiving the collected environmental sound information of the environment in which the robot is located uploaded by the terminal device. The aforementioned sound collection module can be one or more voice collection devices (such as a microphone array) for sound data collection. The aforementioned environmental sound information may include all sounds within a set range where the robot is located (such as an area with a radius of 5m with the robot as the center), such as music, the voice of the target object, and other background sounds.

[0093] In addition, the target sound information can be voice information such as the voice information of the target object, and of course, can also be non-voice information such as music or knocking sounds, which is not specifically limited in this application. In this way, after obtaining voice information and / or non-voice information from the environmental sound information, the voice information and / or non-voice information can be analyzed to obtain information related to the motion control of the robot arm.

[0094] In an optional implementation, after the server obtains the ambient sound information of the environment in which the robotic arm is located, it can perform noise reduction processing on the ambient sound information, filter out the ambient noise included in the ambient sound information, and obtain the ambient sound information after noise reduction processing, and then normalize the ambient sound information after noise reduction processing to obtain the target sound information.

[0095] In this way, the environmental noise included in the collected original sound information (i.e., the environmental sound information) is removed, and the more accurate sound information containing the target sound information is retained, thereby ensuring the accuracy of the subsequent robot arm motion control based on the obtained target sound information. In addition, the target sound information obtained by normalizing the environmental sound information after the noise reduction process can improve the efficiency of the subsequent feature extraction of the target sound information.

[0096] Exemplarily, the server can filter out the ambient noise included in the ambient sound information by a set filtering method (e.g., Kalman filtering, wavelet filtering, etc.) to obtain the ambient sound information after noise reduction processing. Among them, the ambient sound information after noise reduction processing includes target sound information. The above-mentioned normalization processing can normalize the signal amplitude in the ambient sound information after noise reduction processing to a certain range such as [0, 1], and the embodiments of the present application do not specifically limit this.

[0097] S102: Extract features of the target sound information to obtain a first feature set and a second feature set.

[0098] The above-mentioned first feature set may include multiple physical features of the target sound information, and the above-mentioned second feature set may include multiple semantic features of the target sound information. Optionally, the above-mentioned first feature set may include any one or combination of pitch features, volume features, timbre features, speech speed features, duration features and spatial position features corresponding to the target sound information. Among them, the aforementioned duration features can be used to characterize the duration of the target sound information, and the aforementioned spatial position features can be used to characterize the sound source position coordinates corresponding to the target sound information.

[0099] In an alternative implementation, see Figure 2 As shown, when executing step S102, after obtaining the target sound information, the server can extract sound features of the target sound information based on the sound feature extraction methods set for multiple physical features to obtain a first feature set; and perform semantic analysis on the target sound information to obtain a second feature set.

[0100] Exemplarily, the server may use a time-frequency transformation method such as fast Fourier transform (FFT) (i.e., a sound feature extraction method corresponding to the pitch feature) to convert the time domain signal into a frequency domain signal, thereby extracting the spectrum information of the target sound information, and then obtaining the pitch features such as the main frequency and the spectrum range. Assuming that the speech signal of the target object is x(n), the aforementioned FFT transformation formula can be specifically expressed as follows:

[0101]

[0102] Among them, X(k) is the frequency domain signal, x(n) is the time domain signal, that is, the speech signal of the target object, and N is the length of the time domain signal.

[0103] The server may express the volume of the target object by calculating the root mean square (RMS) of the target object's voice signal. Optionally, the aforementioned RMS calculation formula may be specifically expressed as follows:

[0104]

[0105] Among them, V is the root mean square of the speech signal (i.e., volume feature), x(n) is the time domain signal, i.e., the speech signal of the target object, and N is the length of the time domain signal.

[0106] The server can use mel-scale frequency cepstral coefficients (MFCC) to extract timbre features, map the speech signal to the mel-frequency domain, and perform discrete cosine transform (DCT) to obtain MFCC. Optionally, the calculation formula of the mel-frequency corresponding to the speech signal can be specifically expressed as follows:

[0107]

[0108] Among them, m is the Mel frequency corresponding to the speech signal, f is the actual speech frequency of the speech signal, and a is derived through an empirical formula. The purpose is to maintain a certain linear relationship when mapping the frequency to the Mel frequency domain, while conforming to the characteristics of human ear's frequency perception. a can be 2595, and the frequency near b is an area where the human ear is relatively sensitive to frequency changes. Therefore, this value is often used as a benchmark in Mel frequency calculation, and b can be 700.

[0109] The above-mentioned speech speed feature can be determined by the server by counting the number of words in the above-mentioned target sound information within a unit time. The above-mentioned unit time can be 0.5 seconds, 1 second or 2 seconds, etc., and the embodiment of the present application does not make specific limitations on this.

[0110] The above-mentioned duration feature can be that the server performs endpoint detection on the voice signal, determines the start time and end time of the voice signal, and calculates the duration of the voice signal. The above-mentioned duration calculation formula can be specifically expressed as: Δt=t2-t1, where Δt is the duration, t1 is the start time, and t2 is the end time.

[0111] The above-mentioned spatial position features can be determined by the server through the sound source localization technology set for the sound collection module. Take the above-mentioned sound collection module as a microphone array as an example. According to the time difference or phase difference of the voice signals received by different microphone units, the azimuth and elevation angle of the target object, that is, the spatial position features of the target object, are calculated.

[0112] It should be understood that when the server extracts sound features of the target sound information based on the sound feature extraction methods set for multiple physical features and obtains the first feature set, it may be to analyze parameters such as the fundamental frequency and energy of the speech signal to determine the physical features such as pitch features, volume features, timbre features, speaking speed features, duration features, and spatial position features corresponding to the target sound information.

[0113] For another example, the server may use natural language processing (NLP) technology to perform automatic speech recognition (ASR) on the voice signal, convert the target sound information into text information, and thereby perform semantic understanding on the aforementioned text information to obtain the second feature set.

[0114] S103: Based on the first feature set and the second feature set, determine the target performance data of the robot arm and the action instruction set associated with the target performance data.

[0115] In an optional implementation, when executing step S103, the server may refer to Figure 3 The target performance data of the robot arm and the action instruction set associated with the target performance data are determined in the manner shown. The specific process is as follows:

[0116] S301: Determine, from a preset candidate performance data set, target performance data that matches a target semantic feature representing a motion control keyword in a second feature set.

[0117] Exemplarily, the candidate performance data set may include dance tracks of different festivals. Assuming that the target semantic feature representing the motion control keyword in the second feature set is "Spring Festival", the server can determine dance tracks related to "Spring Festival" from the candidate performance data set. Optionally, if there are multiple dance tracks related to "Spring Festival", the dance track with the highest popularity can be used as the target performance data.

[0118] S302: Determine the action parameter subset corresponding to each action in the target performance data based on the mapping relationship between preset features and action parameters, as well as multiple physical features included in the first feature set and at least one semantic feature in the second feature set except the target semantic feature.

[0119] Among them, the mapping relationship between the above-mentioned preset features and action parameters can specifically include: the mapping relationship between pitch features and remote actions of the robotic arm, the mapping relationship between volume features and the amplitude of the robotic arm action, the mapping relationship between timbre features and the type of robotic arm action, the mapping relationship between speech rate features and the speed of the robotic arm action, the mapping relationship between duration features and the duration of the robotic arm action, the mapping relationship between spatial position features and the direction of the robotic arm action, and the mapping relationship between semantic features and robotic arm action combinations.

[0120] Exemplarily, the mapping relationship between the above-mentioned pitch feature and the robot arm movement speed can be: the higher the pitch, the higher the position reached by the robot arm far end; the lower the pitch, the lower the position reached by the robot arm far end. The mapping relationship between the above-mentioned volume feature and the robot arm movement amplitude can be: the larger the volume, the larger the robot arm movement amplitude; the smaller the volume, the smaller the robot arm movement amplitude. The mapping relationship between the above-mentioned timbre feature and the robot arm movement type can be: according to different timbre features, different robot arm movement libraries are called to select the corresponding movement type. The mapping relationship between the above-mentioned speech speed (or music rhythm) feature and the robot arm far end movement can be: the faster the speech speed (or music rhythm), the faster the robot arm movement speed; the slower the speech speed (or music rhythm), the slower the robot arm movement speed. The mapping relationship between the above-mentioned duration feature and the robot arm movement duration can be: the longer the duration, the longer the robot arm movement duration; the shorter the duration, the shorter the robot arm movement duration. The mapping relationship between the above-mentioned spatial position feature and the robot arm movement direction can be: according to the spatial position information of the sound source, the direction of the robot arm movement is determined. The mapping relationship between the above semantic features and the robot arm action combination can be: according to the speech recognition result and semantic understanding (i.e., multiple semantic features included in the second feature set), the corresponding robot arm action combination library is called to generate an action sequence corresponding to the target performance data. For example, if the semantic feature is "forward", the robot arm performs the forward action combination.

[0121] Optionally, the mapping relationship between the above-mentioned preset physical features or semantic features and the robot arm motion parameters can be represented by a mapping function R, and the embodiments of the present application do not specifically limit this. It should be understood that the specific mapping relationships corresponding to different types of target sound information (such as voice information or non-voice information) are different. For example, if the target sound information is voice information or non-voice information containing semantics, the above-mentioned mapping relationship may include: the mapping relationship between pitch features, volume features, timbre features, speech speed features, duration features, spatial position features and semantic features.

[0122] For another example, if the target sound information is non-speech information that does not contain semantics, the above mapping relationship may include: the mapping relationship of volume features, timbre features, speech rate features, duration features, spatial position features, and semantic features. This embodiment of the application does not specifically limit this.

[0123] S303: Generate an action instruction set based on the obtained multiple action parameter subsets.

[0124] In an optional implementation, when executing step S303, after obtaining the action parameter subset corresponding to each action in the target performance data (i.e., obtaining multiple action parameter subsets), the server can perform the following operations for any one of the multiple action parameter subsets, such as the first action parameter subset: based on the conversion relationship between the preset action parameters and the action instructions, generate the first action instruction corresponding to the first action parameter subset; and save the first action instruction in the action instruction set. In this way, the action instruction corresponding to each action in the target performance data can be quickly generated through the conversion relationship between the action parameters and the action instructions.

[0125] Exemplarily, the conversion relationship between the above-mentioned preset action parameters and action instructions can be represented by a mapping function g, which is not specifically limited in the embodiments of the present application.

[0126] It should be understood that each action instruction included in the above action instruction set is generated by a basic action and its corresponding action parameter subset. The basic actions may include but are not limited to: raising hands, turning, swinging, etc., and each basic action has corresponding action parameters such as joint motion trajectory, speed, amplitude, etc. Optionally, the basic action set can be expressed as {A k}, where k represents the sequence number of the basic action, and each basic action A k The action parameter vector can be expressed as: P k =[P k1 , P k2 , ..., P kn ], where P ki Represents the i-th action parameter of the k-th basic action.

[0127] S104: Controlling the robotic arm to display target performance data based on the action instruction set.

[0128] In an optional implementation, after determining the target performance data of the robotic arm and the action instruction set associated with the target performance data based on the first feature set and the second feature set, the server can control the robotic arm to display multiple actions included in the target performance data according to the instruction execution order of the multiple action instructions included in the action instruction set.

[0129] In order to improve the real-time and accuracy of the robot arm control, after each action of the robot arm, the server can also obtain feedback information of the action currently displayed by the robot arm, so as to modify the mapping relationship between the preset features and action parameters and the conversion relationship between the preset action parameters and action instructions based on the aforementioned feedback information, and obtain the modified mapping relationship and the modified conversion relationship; and then adjust the motion instructions at the next moment based on the modified mapping relationship and the modified conversion relationship. The aforementioned feedback information can be used to indicate the completion of the action, as well as the first feature subset and the second feature subset at the next moment adjacent to the current moment.

[0130] Furthermore, the server determines a first modification factor of the mapping relationship and a second modification factor of the conversion relationship based on the feedback information, thereby modifying the mapping relationship based on the first modification factor to obtain the modified mapping relationship, and modifying the conversion relationship based on the second modification factor to obtain the modified conversion relationship.

[0131] For example, assuming that the above feedback information is represented by R b , then the modified mapping relationship can be specifically expressed as: R′=R+ΔR(R b ), where R′ represents the mapping relationship corresponding to the next moment adjacent to the current moment (i.e., the modified mapping relationship), R represents the mapping relationship corresponding to the current moment, ΔR(R b ) represents the above-mentioned first modification factor, and ΔR() represents a function for modifying or adjusting the mapping relationship.

[0132] And, the above modified conversion relationship can be specifically expressed as: ′=+Δ(R b ), where ′ represents the conversion relationship corresponding to the next moment adjacent to the current moment (i.e., the modified conversion relationship), g represents the conversion relationship corresponding to the current moment, Δ(R b ) represents the above-mentioned second modification factor, and Δ() represents a function for modifying or adjusting the conversion relationship.

[0133] Optionally, the feedback information R b Can include: the current moment of action completion status R comp 、Sound change R sound And other relevant information (such as the image of the current environment of the robot arm, etc.). Among them, the sound change R sound A first feature subset and a second feature subset at a next moment adjacent to the current moment may be determined.

[0134] It can be seen that based on the sound information-based robotic arm control method recorded in the above steps S101 to S104, the sound recognition machine technology is combined with the robotic arm control technology, so that the robotic arm can generate performance actions in real time according to the target sound information, thereby improving the interactivity and viewing experience when the robotic arm displays the target performance data; and after collecting the environmental sound information of the environment in which the robotic arm is located, the target sound information included in the environmental sound information can be extracted to obtain a first feature set including multiple physical features and a second feature set including multiple semantic features, thereby determining the target performance data of the robotic arm and the action instruction set associated with the target performance data based on the first feature set and the second feature set, and then controlling the robotic arm to display the target performance data based on the action instruction set. It can be seen that according to the target sound information, the real-time adjustment and dynamic response of the robotic arm action can be realized, thereby increasing the interactivity and flexibility in the robotic arm control process.

[0135] Further, based on the same technical concept, the embodiment of the present application provides a robotic arm control device based on sound information, and the robotic arm control device based on sound information is used to implement the above method flow of the embodiment of the present application. Figure 4 As shown, the robot arm control device 400 based on sound information may include: an information acquisition module 401, a feature extraction module 402, an instruction determination module 403 and an instruction control module 404, wherein:

[0136] The information acquisition module 401 is used to acquire target sound information from the collected environmental sound information of the environment where the robot arm is located; the target sound information includes motion control keywords for controlling the motion of the robot arm;

[0137] A feature extraction module 402 is used to extract features from the target sound information to obtain a first feature set and a second feature set; wherein the first feature set includes multiple physical features of the target sound information, and the second feature set includes multiple semantic features of the target sound information;

[0138] An instruction determination module 403 is used to determine the target performance data of the robot arm and the action instruction set associated with the target performance data based on the first feature set and the second feature set;

[0139] The instruction control module 404 is used to control the robot arm to display target performance data based on the action instruction set.

[0140] In an optional embodiment, when acquiring target sound information from environmental sound information, the information acquisition module 401 is specifically used to:

[0141] Performing noise reduction processing on the ambient sound information, filtering out the ambient noise included in the ambient sound information, and obtaining the ambient sound information after the noise reduction processing;

[0142] The ambient sound information after noise reduction is normalized to obtain the target sound information.

[0143] In an optional embodiment, when extracting features from the target sound information to obtain the first feature set and the second feature set, the feature extraction module 402 is specifically configured to:

[0144] Based on the sound feature extraction methods respectively set for the multiple physical features, extract the sound features of the target sound information to obtain a first feature set; and

[0145] Perform semantic analysis on the target sound information to obtain a second feature set.

[0146] In an optional embodiment, when determining the target performance data of the robot arm and the action instruction set associated with the target performance data based on the first feature set and the second feature set, the instruction determination module 403 is specifically used to:

[0147] Determining, from a preset candidate performance data set, target performance data that matches a target semantic feature representing a motion control keyword in a second feature set;

[0148] Determine an action parameter subset corresponding to each action in the target performance data based on a mapping relationship between preset features and action parameters, and a plurality of physical features included in the first feature set and at least one semantic feature in the second feature set except the target semantic feature;

[0149] An action instruction set is generated based on the obtained multiple action parameter subsets.

[0150] In an optional embodiment, when generating the action instruction set based on the obtained multiple action parameter subsets, the instruction determination module 403 is specifically used to:

[0151] For multiple action parameter subsets, perform the following operations respectively:

[0152] Based on the conversion relationship between the preset action parameters and the action instructions, generate a first action instruction corresponding to the first action parameter subset; wherein the first action parameter subset is any one of the multiple action parameter subsets;

[0153] The first action instruction is saved in the action instruction set.

[0154] In an optional embodiment, when controlling the mechanical arm to display target performance data based on the action instruction set, the instruction control module 404 is specifically used to:

[0155] Controlling the robot arm to display multiple actions included in the target performance data according to the instruction execution order of multiple action instructions included in the action instruction set;

[0156] After each action of the robot arm, the following operations are performed:

[0157] Obtain feedback information of the action performed by the robot arm at the current moment; the feedback information is used to indicate the completion status of the action, as well as the first feature subset and the second feature subset of the next moment adjacent to the current moment;

[0158] Modifying the mapping relationship between the preset features and the action parameters and the conversion relationship between the preset action parameters and the action instructions based on the feedback information to obtain a modified mapping relationship and a modified conversion relationship;

[0159] The motion instruction at the next moment is adjusted based on the modified mapping relationship and the modified conversion relationship.

[0160] In an optional embodiment, when modifying the mapping relationship between the preset features and the action parameters and the conversion relationship between the preset action parameters and the action instructions based on the feedback information, the instruction control module 404 is specifically used to:

[0161] Determine a first modification factor of the mapping relationship and a second modification factor of the conversion relationship based on the feedback information;

[0162] The mapping relationship is modified based on the first modification factor, and the conversion relationship is modified based on the second modification factor.

[0163] Based on the description of the above method embodiment and device embodiment, the exemplary embodiment of the present invention further provides an electronic device, including: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method according to the embodiment of the present invention when executed by the at least one processor.

[0164] An embodiment of the present application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute a method according to an embodiment of the present application.

[0165] An embodiment of the present application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute a method according to an embodiment of the present application.

[0166] See also Figure 5As shown, the structured block diagram of the electronic device 500 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent the computer device of various forms of digital electronics, such as, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described herein and / or required.

[0167] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0168] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 may be any type of device capable of inputting information to the electronic device 500, and the input unit 506 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 507 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 may include but is not limited to a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a worldwide interoperability for microwave access (WiMax) device, a cellular communication device, and / or the like.

[0169] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various AI computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above.

[0170] For example, in some embodiments, the above-mentioned robot arm control method based on sound information can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. In some embodiments, the computing unit 501 can be configured to execute the above-mentioned robot arm control method based on sound information in any other appropriate manner (for example, by means of firmware).

[0171] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0172] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0173] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0174] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tub (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0176] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0177] Furthermore, it should be understood that what is disclosed above is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope covered by the present application.

Claims

1. A robotic arm control method based on sound information, characterized in that: include: Obtain target sound information from the collected environmental sound information of the environment in which the robot arm is located; The target sound information includes motion control keywords for controlling the motion of the robotic arm; Extracting features of the target sound information to obtain a first feature set and a second feature set; wherein the first feature set includes a plurality of physical features of the target sound information, and the second feature set includes a plurality of semantic features of the target sound information; Determining target performance data of the robotic arm and a motion instruction set associated with the target performance data based on the first feature set and the second feature set; The robotic arm is controlled to display the target performance data based on the action instruction set.

2. The method according to claim 1, characterized in that The step of obtaining target sound information from the collected ambient sound information of the environment in which the robot arm is located includes: Performing noise reduction processing on the ambient sound information, filtering out ambient noise included in the ambient sound information, and obtaining ambient sound information after noise reduction processing; The ambient sound information after the noise reduction process is normalized to obtain the target sound information.

3. The method according to claim 1, characterized in that The extracting features of the target sound information to obtain a first feature set and a second feature set includes: Based on the sound feature extraction methods respectively set for the multiple physical features, extracting the sound features of the target sound information to obtain the first feature set; and Perform semantic analysis on the target sound information to obtain the second feature set.

4. The method according to claim 1, 2 or 3, characterized in that: The first feature set includes any one or a combination of pitch features, volume features, timbre features, speaking speed features, duration features and spatial position features; wherein the duration feature is used to characterize the duration of the target sound information, and the spatial position feature is used to characterize the sound source position coordinates corresponding to the target sound information.

5. The method according to claim 1, 2 or 3, characterized in that: The step of determining the target performance data of the robot arm and the action instruction set associated with the target performance data based on the first feature set and the second feature set includes: Determining, from a preset set of candidate performance data, the target performance data that matches the target semantic feature representing the motion control keyword in the second feature set; Determine an action parameter subset corresponding to each action in the target performance data based on a mapping relationship between preset features and action parameters, the multiple physical features included in the first feature set, and at least one semantic feature in the second feature set except the target semantic feature; The action instruction set is generated based on the obtained multiple action parameter subsets.

6. The method according to claim 5, characterized in that The step of generating the action instruction set based on the obtained plurality of action parameter subsets comprises: For the multiple action parameter subsets, the following operations are performed respectively: Based on the conversion relationship between the preset action parameters and the action instructions, generate a first action instruction corresponding to the first action parameter subset; wherein the first action parameter subset is any one of the multiple action parameter subsets; The first action instruction is saved in the action instruction set.

7. The method according to claim 1, 2 or 3, characterized in that: The controlling the mechanical arm to display the target performance data based on the action instruction set includes: Controlling the robot arm to display multiple actions included in the target performance data according to the instruction execution order of multiple action instructions included in the action instruction set; Wherein, after each action performed by the robotic arm, the following operations are performed: Acquire feedback information of the action performed by the robot arm at the current moment; the feedback information is used to indicate the completion status of the action, as well as the first feature subset and the second feature subset at the next moment adjacent to the current moment; Modifying the mapping relationship between the preset features and the action parameters and the conversion relationship between the preset action parameters and the action instructions based on the feedback information to obtain a modified mapping relationship and a modified conversion relationship; The motion instruction at the next moment is adjusted based on the modified mapping relationship and the modified conversion relationship.

8. The method according to claim 7, characterized in that The modifying, based on the feedback information, the mapping relationship between the preset features and the action parameters and the conversion relationship between the preset action parameters and the action instructions comprises: Determine a first modification factor of the mapping relationship and a second modification factor of the conversion relationship based on the feedback information; The mapping relationship is modified based on the first modification factor, and the conversion relationship is modified based on the second modification factor.

9. A robotic arm control device based on sound information, characterized in that: include: An information acquisition module is used to acquire target sound information from the collected environmental sound information of the environment in which the robot arm is located; The target sound information includes motion control keywords for controlling the motion of the robotic arm; A feature extraction module, configured to extract features from the target sound information to obtain a first feature set and a second feature set; wherein the first feature set includes a plurality of physical features of the target sound information, and the second feature set includes a plurality of semantic features of the target sound information; An instruction determination module, used for determining target performance data of the robot arm and an action instruction set associated with the target performance data based on the first feature set and the second feature set; An instruction control module is used to control the robotic arm to display the target performance data based on the action instruction set.

10. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 8.

Images