Robot arm device and robot

Abstract

The embodiment of the invention provides a mechanical arm device and a robot, and the device comprises a voice recognition unit which is used for collecting and carrying out the semantic recognition of a voice command, and outputting an instruction semantic recognition result; the mechanical arm unit comprises a mechanical arm, and the mechanical arm unit is used for controlling the action of the mechanical arm according to the mechanical arm control signal; the mechanical arm control SOC unit is connected with the two units and comprises a CPU module designed based on a RiscV instruction set, and the CPU module stores pre-constructed voice processing and mechanical arm control instruction packets and processes instructions through a five-level assembly line architecture; the CPU module is used for controlling the voice recognition unit to collect a target voice command through the voice processing instruction packet so as to obtain a target instruction semantic recognition result; and determining a first target mechanical arm control instruction from the mechanical arm control instruction packet according to the target instruction semantic recognition result, and executing the first target mechanical arm control instruction to generate a mechanical arm control signal. The embodiment of the invention can improve the real-time performance of the action of the mechanical arm.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of mechanical arm, in particular to a mechanical arm device and a robot. BACKGROUND

[0002] In related mechanical arm technologies controlled by voice, the SOC for controlling the mechanical arm generally uses a general instruction set to realize the action of the mechanical arm, so when the SOC controls the mechanical arm through the general instruction set, a large number of instruction execution processes (including instruction fetching, decoding, memory access, etc.) are experienced, so that one mechanical arm action may need to be realized through the execution of multiple general instructions, which makes there be a delay between the input voice command and the mechanical arm action that can be clearly felt, and the real-time performance of the mechanical arm action cannot be guaranteed. SUMMARY

[0003] Embodiments of the present application provide a mechanical arm device and a robot, aiming to reduce the delay between the input voice command and the mechanical arm action, so as to improve the real-time performance of the mechanical arm action.

[0004] In a first aspect, embodiments of the present application provide a mechanical arm device, comprising: a voice recognition unit, configured to collect a voice command and perform semantic recognition on the voice command, and output an instruction semantic recognition result; a mechanical arm unit, comprising a mechanical arm, the mechanical arm unit being configured to control the action of the mechanical arm according to a mechanical arm control signal; a mechanical arm control SOC unit, connected with the voice recognition unit and the mechanical arm unit respectively, the mechanical arm control SOC unit comprising a CPU module designed based on an extension of a RiscV instruction set, the CPU module storing a pre-constructed voice processing instruction package and a mechanical arm control instruction package, and the CPU module being configured to perform the following steps through a five-stage pipeline architecture: controlling the voice recognition unit to collect a target voice command through the voice processing instruction package, so as to obtain a target instruction semantic recognition result from the voice recognition unit; determining a first target mechanical arm control instruction from the mechanical arm control instruction package according to the target instruction semantic recognition result; executing the first target mechanical arm control instruction to generate a mechanical arm control signal.

[0005] In an embodiment, the mechanical arm device further comprises: a human-computer interaction controller, connected with the mechanical arm control SOC unit, the human-computer interaction controller being configured to output a target operation command according to a human-computer interaction behavior; the CPU module is further configured to: determining a second target robot control instruction from the robot control instruction package according to the target operation command; executing the second target robot control instruction to generate the robot control signal.

[0006] In an embodiment, the robot device further comprises: an image acquisition unit connected with the robot control SOC unit, the image acquisition unit being configured to acquire an environment image of the robot; The determining a first target robot control instruction from the robot control instruction package according to the target instruction semantic recognition result comprises: In a case where the semantic content of the target voice command contains content of applying an image, invoking the environment image according to the target instruction semantic recognition result; identifying the environment image, and determining the first target robot control instruction from the robot control instruction package according to an obtained image recognition result.

[0007] In an embodiment, the determining a second target robot control instruction from the robot control instruction package according to the target operation command comprises: In a case where the target operation command contains content of applying an image, determining the second target robot control instruction from the robot control instruction package according to the target operation command and the environment image.

[0008] In an embodiment, the CPU module is further configured to: In a case where any two of the target instruction semantic recognition result, the target operation command and the environment image wait for processing of the other one, determining a target processing order between the target instruction semantic recognition result, the target operation command and the environment image; processing the target instruction semantic recognition result, the target operation command and the environment image according to the target processing order, and determining a third target robot control instruction from the robot control instruction package according to a processing result; executing the third target robot control instruction to generate the robot control signal.

[0009] In an embodiment, the robot unit comprises: a robot action data acquisition module installed on the robot and connected with the robot control SOC unit, the robot action data acquisition module being configured to acquire robot action data of the robot; The executing the first target robot control instruction to generate a robot control signal comprises: According to the first target mechanical arm control instruction, the mechanical arm action data is subjected to control data operation to obtain first mechanical arm control data; The mechanical arm control signal is generated according to the first mechanical arm control data.

[0010] In an embodiment, the CPU module is further configured to: After the mechanical arm control signal is generated, the mechanical arm action data collected by the mechanical arm action data collection module during the action of the mechanical arm according to the mechanical arm control signal is acquired; The second mechanical arm control data is obtained by performing control data operation according to the mechanical arm action data. The mechanical arm control signal is updated according to the second mechanical arm control data.

[0011] In an embodiment, the mechanical arm action data collection module comprises at least one of the following: An inertial measurement module is installed on the mechanical arm and connected to the mechanical arm control SOC unit, and the inertial measurement module is configured to collect inertial data of the mechanical arm. A force feedback module is installed on the mechanical arm and connected to the mechanical arm control SOC unit, and the force feedback module is configured to collect force data of the mechanical arm. The mechanical arm action data comprises at least one of the following: The inertial data collected by the inertial measurement module. The force data collected by the force feedback module.

[0012] In an embodiment, the mechanical arm device further comprises: A display unit connected to the mechanical arm control SOC unit, and the display unit is configured to display the action of the mechanical arm according to the mechanical arm control signal.

[0013] In a second aspect, the embodiments of the present application provide a robot comprising the mechanical arm device as described in the first aspect.

[0014] The embodiment of the application provides a mechanical arm device and a robot, the device comprises a voice recognition unit, which is used for collecting a voice command and performing semantic recognition on the voice command to output an instruction semantic recognition result; a mechanical arm unit comprising a mechanical arm, the mechanical arm unit is used for controlling the action of the mechanical arm according to a mechanical arm control signal; a mechanical arm control SOC unit connected with the voice recognition unit and the mechanical arm unit respectively, the mechanical arm control SOC unit comprises a CPU module designed based on a RiscV instruction set, the CPU module stores a pre-constructed voice processing instruction package and a mechanical arm control instruction package, and the CPU module is used for executing the following steps through a five-stage pipeline architecture: controlling the voice recognition unit to collect a target voice command through the voice processing instruction package, so as to obtain a target instruction semantic recognition result from the voice recognition unit; determining a first target mechanical arm control instruction from the mechanical arm control instruction package according to the target instruction semantic recognition result; and executing the first target mechanical arm control instruction to generate a mechanical arm control signal. Since the voice processing instruction package is constructed by RiscV instruction set extension, the CPU module can control the voice recognition unit through the instructions in the voice processing instruction package, so that the number of general instructions used in the process of outputting the target instruction semantic recognition result can be reduced, and then the number of general instructions executed in the process of outputting the target instruction semantic recognition result can be reduced; on this basis, since the mechanical arm control instruction package is also constructed by RiscV instruction set extension, the number of general instructions used in the subsequent mechanical arm control signal generation process through the first target mechanical arm control instruction in the mechanical arm control instruction package can be reduced, and then the number of general instructions executed in the process of generating the mechanical arm control signal can be reduced; since the execution number of general instructions is reduced in the above two processes, and the five-stage pipeline architecture is used for instruction processing, the delay caused by the large number of general instructions in the process of controlling the mechanical arm through the voice can be greatly reduced, so that the real-time performance of the mechanical arm action can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a structural schematic diagram of a mechanical arm device provided by the embodiment of the application; Figure 2 is a flowchart of steps executed by the mechanical arm control SOC unit according to an embodiment of the application; Figure 3 is a structural schematic diagram of a mechanical arm device comprising the internal structure of the mechanical arm control SOC unit according to the embodiment of the application; Figure 4 is a five-stage pipeline structure schematic diagram according to the embodiment of the application; Figure 5 is Figure 4 the processing flowchart of the pipeline architecture in Figure 6 is a flowchart of steps performed by the mechanical arm control SOC unit according to another embodiment of the present application; Figure 7 is a flowchart of steps performed by the mechanical arm control SOC unit according to another embodiment of the present application; Figure 8 is a flowchart of steps performed by the mechanical arm control SOC unit according to another embodiment of the present application.

[0016] Reference signs: Voice recognition unit 1, mechanical arm unit 2, mechanical arm 21, mechanical arm action data acquisition module 22, mechanical arm control SOC unit 3, human-computer interaction controller 4, image acquisition unit 5, display unit 6, state prompting unit 7. DETAILED DESCRIPTION

[0017] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0018] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0019] In the embodiments of the present application, the words "further", "exemplarily" or "optionally" are used to represent as an example, illustration or description, and should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. The use of "further", "exemplarily" or "optionally" is intended to present the relevant concept in a specific manner.

[0020] In the related technology of mechanical arm controlled by voice, the SOC for controlling the mechanical arm generally uses a general instruction set to realize the action of the mechanical arm, so when the SOC controls the mechanical arm through the general instruction set, a large number of instruction execution processes (including instruction fetching, decoding, memory access, etc.) will be experienced, so that one mechanical arm action may need to be realized through the execution of multiple general instructions, which makes there is a delay that can be clearly felt between the input voice command and the mechanical arm action, and the real-time performance of the mechanical arm action cannot be guaranteed.

[0021] In order to reduce the delay between the input voice command and the action of the mechanical arm, and improve the real-time performance of the mechanical arm action, the embodiment of the application provides a mechanical arm device and a robot. The device comprises a voice recognition unit, which is used to collect voice commands and perform semantic recognition on the voice commands, and output instruction semantic recognition results; a mechanical arm unit, which comprises a mechanical arm, and is used to control the action of the mechanical arm according to a mechanical arm control signal; and a mechanical arm control SOC unit connected with the voice recognition unit and the mechanical arm unit respectively, wherein the mechanical arm control SOC unit comprises a CPU module designed based on a RiscV instruction set, and the CPU module stores a pre-constructed voice processing instruction package and a mechanical arm control instruction package, and is used to execute the following steps through a five-stage pipeline architecture: controlling the voice recognition unit to collect target voice commands through the voice processing instruction package, so as to obtain target instruction semantic recognition results from the voice recognition unit; determining a first target mechanical arm control instruction from the mechanical arm control instruction package according to the target instruction semantic recognition results; and executing the first target mechanical arm control instruction to generate a mechanical arm control signal. Since the voice processing instruction package is constructed by RiscV instruction set extension, the CPU module can control the voice recognition unit through the instructions in the voice processing instruction package, so as to reduce the number of general instructions used in the process of outputting the target instruction semantic recognition results, and further reduce the number of general instructions executed in the process of outputting the target instruction semantic recognition results. On this basis, since the mechanical arm control instruction package is also constructed by RiscV instruction set extension, the number of general instructions used in the subsequent mechanical arm control signal generation process through the first target mechanical arm control instruction in the mechanical arm control instruction package can be reduced, and the number of general instructions executed in the process of generating the mechanical arm control signal can be further reduced. Since the execution number of general instructions is reduced in the above two processes, and the five-stage pipeline architecture is used for instruction processing, the delay caused by the large number of general instructions in the process of controlling the mechanical arm through voice can be greatly reduced, so as to improve the real-time performance of the mechanical arm action.

[0022] Referring to Figure 1 , Figure 1 The structure of the mechanical arm device provided by the embodiment of the application is shown. In the embodiment of the application, the mechanical arm device can comprise a voice recognition unit 1, a mechanical arm unit 2 and a mechanical arm control SOC unit 3.

[0023] The voice recognition unit 1 can be used to collect voice commands and perform semantic recognition on the voice commands to output instruction semantic recognition results. The mechanical arm unit 2 can include a mechanical arm 21, and the mechanical arm unit 2 can be used to control the action of the mechanical arm 21 according to a mechanical arm control signal. The mechanical arm control SOC unit 3 is connected to the voice recognition unit 1 and the mechanical arm unit 2, respectively, and the mechanical arm control SOC unit 3 can include a CPU module designed based on a RiscV instruction set, which stores pre-constructed voice processing instruction packages and mechanical arm control instruction packages.

[0024] Referring to Figure 2 , the CPU module can be used to perform the following steps through a five-stage pipeline architecture.

[0025] Step 110: Collect target voice commands through the voice processing instruction package to obtain target instruction semantic recognition results from the voice recognition unit; Step 120: Determine a first target mechanical arm control instruction from the mechanical arm control instruction package according to the target instruction semantic recognition results; Step 130: Execute the first target mechanical arm control instruction to generate a mechanical arm control signal.

[0026] In an embodiment, the voice recognition unit 1 is a voice interaction hardware that integrates a pre-trained semantic recognition model. The voice recognition unit 1 can be a voice module that performs local semantic recognition, such as an LD3320, an ASR PRO series, etc.

[0027] In an embodiment, in the process of performing semantic recognition on the voice commands and outputting instruction semantic recognition results, the voice recognition unit 1 can call a local semantic recognition model to perform semantic recognition on the collected voice commands and output semantic recognition results. The voice command is a natural language command input by the user through the voice recognition unit 1 to control the mechanical arm 21 to perform the action specified by the user, such as "put the green cup on the blue box" or "remove the green cup from the blue box", etc. without limitation.

[0028] In an embodiment, in the process of calling the semantic recognition model to perform semantic recognition on the voice commands and outputting instruction semantic recognition results, the voice recognition unit 1 can first extract semantic features from the voice commands through a voice feature library, then compare the semantic features with the voice feature library stored in its own cache to determine and output the instruction semantic recognition results corresponding to the semantic features. The voice feature library is stored on a storage medium in the mechanical arm control SOC unit 3, but when the mechanical arm device is started, the mechanical arm control SOC unit can transmit the voice feature library to the internal cache of the voice recognition unit 1, so that the voice recognition unit 1 can quickly call the data in the voice feature library.

[0029] In an embodiment, the voice feature library can add semantic features of voice commands in different scenarios by replacing or adding, so that the voice recognition unit 1 can recognize voice commands in different application scenarios such as medical surgery scenarios and industrial manufacturing scenarios.

[0030] In an embodiment, the semantic features of the voice feature library can be updated by deploying a machine learning model in the robot control SOC unit 3 to learn historical voice commands, or by setting a communication module based on the robot control SOC unit 3 to return the input instructions to the cloud and receive the semantic features parsed from the cloud, and the like. Therefore, the voice feature library can store semantic features of voice commands in different accents and semantic features of voice commands in noisy environments, so as to adapt to different complex application environments and help the voice recognition unit 1 to output more accurate semantic recognition results. The machine learning model for updating the data of the voice feature library can be a quantized convolutional neural network model QCNN, a light recurrent neural network RNN, and the like, which is not limited here.

[0031] In an embodiment, the robot control SOC unit 3 refers to an SOC hardware unit with a RiscV core, which includes the CPU module mentioned above. The CPU module based on the RiscV instruction set refers to an integrated central control module that meets the usage rules of the RiscV instruction set. Referring to Figure 3 , the robot control SOC unit 3 can include a CPU module (shown as VexRiscV in Figure 3 , which belongs to one of RiscV) and peripheral interfaces of various interface protocols (such as GPIO interface, SPI interface, DPI interface, SPI interface, I 2 The CPU module can include a JTAG debugging interface for external debugging, an on-chip memory BSRAM, a register group Reg (including a variety of registers such as a status register, a data register, and a control register), a write buffer hardware WBU, a memory access hardware mem, an instruction cache Icache, a data cache Dcache, an instruction pipeline unit IFU, an instruction decoding unit IDU, and an execution unit EXU, an AMBA AXI4 protocol-based crossbar AXI4Cross Bar, an APB protocol-based bridge APB3 Bridge, and a peripheral interface. BSRAM, Icache, Dcache, APB3Bridge, mem, and Reg (the connection between mem, Reg, and AXI4 Cross Bar is not shown in the figure) Figure 3The AXI4 Cross Bar is connected with the Icache, the Dcache, the Reg, the mem, the APB3 Bridge, the IFU, the IDU, the EXU, and the WBU. The Icache is connected with the IFU, the Dcache is connected with the IDU, the Reg is connected with the WBU, the mem is connected with the EXU, the IFU is connected with the IDU, the IDU is connected with the EXU, the EXU is connected with the WBU, and the APB3 Bridge is connected with the peripherals of various interface protocols.

[0032] In the case of no external data trigger (such as the mechanical device is powered on), the CPU module can follow the established flow to fetch instructions from the Icache through the IFU, the IDU decodes (or translates) the instructions fetched by the IFU to make the instructions into a form that can be executed by the EXU, the EXU performs instruction execution and obtains operation data or execution results. In the process of instruction execution by the EXU, if data needs to be written into the memory, the data is stored to the Dcache, and the Dcache writes back to the BSRAM through the AXI4 Cross Bar; if data needs to be read from the BSRAM, the Dcache is first accessed to the most recent data, if the corresponding data is not queried, the data is read from the BSRAM through the mem and the AXI4 Cross Bar. After that, if at least one of the data accessed from the BSRAM and the execution result obtained by the EXU needs to be written into the Reg, it is written into the Reg through the WBU, and the Reg is output to the corresponding hardware device through the AXI4 Cross Bar, the APB3 Bridge and the peripheral interface. For example, the voice feature library mentioned above can be written into the voice recognition unit 1 in this way.

[0033] In the case of external data triggering (such as the voice recognition unit 1 inputting a semantic recognition result), the external interface transmits the incoming data through the APB3 Bridge for protocol conversion, and writes the data into the Reg through the AXI4 Cross Bar, then the IFU fetches instructions from the Icache, the IDU decodes the instructions fetched by the IFU to make the instructions executable by the EXU, and the EXU performs the instructions to obtain data or execution results. During the execution of the instructions by the EXU, if the data needs to be written into the BSRAM, the data is stored in the Dcache, and the Dcache writes the data back to the BSRAM through the AXI4 Cross Bar; if the data needs to be read from the BSRAM, the Dcache is first accessed to obtain the latest data, and if the corresponding data is not found, the data is read from the BSRAM through the mem and the AXI4 Cross Bar. After that, if at least one of the data read from the BSRAM and the execution result obtained by the execution unit EXU needs to be written into the Reg, the data is written into the Reg through the WBU, and the Reg is output to the corresponding hardware device through the AXI4 Cross Bar, the APB3 Bridge and the peripheral interface. For example, the above-mentioned mechanical arm control signal can control the mechanical arm 21 in the mechanical arm unit 2 to move in this way.

[0034] Since the mechanical arm control SOC unit 3 is a RiscV kernel-based SOC unit, it has the function of developing other units, improves the functional expansion of the mechanical arm device, and reduces the development cycle.

[0035] In an embodiment, the CPU module described above can be integrated in a FPGA-based running platform. The FPGA-based running platform can be Tang Mega 138K Pro Dock, Xilinx Zynq-7000 series, etc., which is not limited here.

[0036] In an embodiment, the Reg includes a program counter PC, and the PC records a PC value, which is used by the IFU to obtain instructions mapped by the PC value from the Icache. The PC value in the PC is changed in the case of no external data triggering and the case of external data triggering, so that the CPU module can execute instructions in various cases.

[0037] In an embodiment, referring to Figure 4 and Figure 5 , the IFU, IDU, EXU, mem and WBU form the five-stage pipeline architecture described above. The IFU fetches the first instruction inst1 in the first stage (cycle1); the IDU decodes the instructions fetched by the IFU in the second stage (cycle2); the EXU performs the instructions in the third stage (cycle3); the mem reads data from the BSRAM in the fourth stage (cycle4); and the WBU writes data into the Reg in the fifth stage (cycle5). Figure 5 Figure 5 ​The first instruction inst1 is decoded in the second stage (cycle 2 as shown in the figure), at which time the IFU can fetch the second instruction inst2; the execution unit EXU performs in the third stage (cycle 3 as shown in the figure), at which time the IDU can decode the second instruction inst2 and the IFU can fetch the third instruction inst3; if there is a memory requirement, the mem can perform memory access for the first instruction inst1 in the fourth stage (cycle 4 as shown in the figure), at which time the EXU can execute the second instruction inst2, the IDU can decode the third instruction inst3, and the IFU can fetch the fourth instruction inst4; if there is a write requirement, the WBU can perform write in the fifth stage (cycle 5 as shown in the figure), at which time, if the second instruction inst2 has a memory requirement, the mem can perform memory access for the second instruction inst2 in the fifth stage, the EXU can execute the third instruction inst3, the IDU can decode the fourth instruction inst4, and the IFU can fetch the fifth instruction inst5, and so on. By dividing the execution process of the instructions into multiple stages and allowing different instructions to be executed in parallel in different stages, a new instruction enters the pipeline and an instruction completes execution and exits the pipeline every time a stage is experienced, so that the throughput of the processor can be significantly improved. Figure 5 Figure 5 The first instruction inst1 is decoded in the second stage (cycle 2 as shown in the figure), at which time the IFU can fetch the second instruction inst2; the execution unit EXU performs in the third stage (cycle 3 as shown in the figure), at which time the IDU can decode the second instruction inst2 and the IFU can fetch the third instruction inst3; if there is a memory requirement, the mem can perform memory access for the first instruction inst1 in the fourth stage (cycle 4 as shown in the figure), at which time the EXU can execute the second instruction inst2, the IDU can decode the third instruction inst3, and the IFU can fetch the fourth instruction inst4; if there is a write requirement, the WBU can perform write in the fifth stage (cycle 5 as shown in the figure), at which time, if the second instruction inst2 has a memory requirement, the mem can perform memory access for the second instruction inst2 in the fifth stage, the EXU can execute the third instruction inst3, the IDU can decode the fourth instruction inst4, and the IFU can fetch the fifth instruction inst5, and so on. By dividing the execution process of the instructions into multiple stages and allowing different instructions to be executed in parallel in different stages, a new instruction enters the pipeline and an instruction completes execution and exits the pipeline every time a stage is experienced, so that the throughput of the processor can be significantly improved. Figure 5

[0038] In an embodiment, the IFU is internally provided with branch instruction prediction hardware such as a branch target buffer BTB, a predictor, etc. The BTB can store addresses of recently executed branch instructions, target addresses of the branch instructions, and 2-bit history information. When the IFU fetches instructions from the instruction cache Icache according to the PC value recorded in the PC, the PC value is sent to the BTB for query and to the predictor for branch prediction. If the BTB hits, the hit target address is directly returned; at the same time of query, the predictor also performs branch prediction according to the 2-bit history information indexed by the PC value. If the branch prediction is successful, the fetched instruction is continuously executed; if the branch prediction fails, the prediction result replaces the corresponding entry in the BTB. By dynamically predicting branch instructions through 2-bit history information, the branch direction and the branch target address can be predicted at the same time, so that the accuracy of branch prediction can be improved, which helps to improve the instruction fetching performance of the IFU.

[0039] It should be noted that the CPU module follows the above-mentioned architecture embodiment in the process of executing the above-mentioned steps and the steps to be described below.

[0040] ​​In an embodiment, the mechanical arm control SOC unit 3 can be connected with hardware devices including the voice recognition unit 1 and the mechanical arm unit 2 through a plurality of peripheral interfaces. The interface protocols of each hardware device can be the same or different, for example, the voice recognition unit 1 can be connected with the mechanical arm control SOC unit 3 through a GPIO interface, the mechanical arm unit 2 can be connected with the mechanical arm control SOC unit 3 through a UART interface, and the like.

[0041] In an embodiment, the pre-constructed voice processing instruction package refers to an instruction architecture rule based on a RiscV instruction set and a hardware running mode of the voice recognition unit 1, and is used to control a custom execution instruction set of the voice recognition unit 1. The pre-constructed mechanical arm control instruction package refers to an instruction architecture rule of a RiscV instruction set and a hardware running mode of the mechanical arm unit 2, and is used to control a custom execution instruction set of the mechanical arm unit 2. By constructing the voice processing instruction package and the mechanical arm control instruction package based on the RiscV instruction set, the number of instructions used in the execution of the voice command and the mechanical arm operation can be reduced, thereby improving the throughput of the instructions and helping to improve the real-time performance of the action of the mechanical arm 21.

[0042] In an embodiment, the BSRAM can store a plurality of action behavior parameters of the mechanical arm, and the mechanical arm control instruction package can include a mechanical arm control instruction for calling a preset action behavior parameter, thereby reducing the calculation amount of the CPU module and improving the action execution efficiency of the mechanical arm 21.

[0043] In an embodiment, the control of the mechanical arm 21 can apply to a PID algorithm, a rudder control algorithm (such as Kalman filtering), and the like. Some mechanical arm control instructions in the mechanical arm control instruction package can integrate the calculation steps of these algorithms, and the calling of general instructions can be reduced when these algorithms are executed. Thus, the use of general instructions can be reduced, the throughput of the instructions can be improved, and the real-time performance of the action of the mechanical arm 21 can be improved.

[0044] In an embodiment, the voice recognition unit 1 is controlled by the voice processing instruction package to collect a target voice command, so as to obtain a target instruction semantic recognition result from the voice recognition unit 1, which refers to determining an instruction PC value of a target voice processing instruction, determining the target voice processing instruction from the voice processing instruction package according to the mapped instruction PC value, and controlling the voice recognition unit 1 through the target voice processing instruction, so that the voice recognition unit 1 can collect and perform semantic recognition on a target voice command issued by a user, and output a target instruction semantic recognition result. The target voice processing instruction is a hardware instruction, which refers to the first voice processing instruction determined in the voice processing instruction package in the process of starting and running the voice recognition unit 1.

[0045] In an embodiment, the manner of determining the instruction PC value of the target voice processing instruction can be various, such as the robot arm device is provided with multiple operation modes, each operation mode corresponds to a different instruction PC value of the target voice processing instruction, and the instruction PC value is determined according to the current operation mode; or the robot arm device starts the voice recognition unit 1 in a unique manner, that is, the instruction PC value of the target voice processing instruction is fixed, and the like, which is not limited here.

[0046] In an embodiment, the manner of controlling the voice recognition unit 1 by the target voice processing instruction can be various, such as the target voice processing instruction is an instruction that does not involve the calling of other instructions, and after the processing of the above-mentioned pipeline structure, the execution result is written into Reg, Reg outputs the written value to the voice recognition unit 1, so that the voice recognition unit 1 completes the running configuration and enters the running state; or the target voice processing instruction is an instruction involving the calling of other instructions, and after the processing of the above-mentioned pipeline structure, part of the execution result is written into Reg and the instruction PC value in PC is modified, so that the IFU calls other instructions (which can be instructions in the voice processing instruction package or other instructions) for execution, and the final execution result is also written into the register group, and then Reg outputs the written value to the voice recognition unit 1, so that the voice recognition unit 1 completes the running configuration and enters the running state, and the like, which is not limited here.

[0047] In an embodiment, determining the first target robot arm control instruction from the robot arm control instruction package according to the target instruction semantic recognition result refers to determining the instruction PC value mapped by the instruction semantic recognition result, determining the instruction according to the mapped instruction PC value, and determining the operation of the first target robot arm control instruction in the robot arm control instruction package. Wherein, the first target robot arm control instruction mentioned here and the second target robot arm control instruction and the third target robot arm control instruction mentioned below all refer to hardware instructions for determining the running parameters of the robot arm 21.

[0048] In an embodiment, the instruction PC value mapped by the instruction semantic recognition result can directly or indirectly map the first target robot control instruction, which is not specifically limited here. For example, assuming that a PC value is set for a specific target instruction semantic recognition result, direct mapping refers to the case where the PC value is set to be directly mapped to the instruction semantic recognition result, so that the first target robot control instruction can be directly determined in the robot control instruction package through the mapped instruction PC value; indirect mapping refers to the case where the PC value is different from the instruction PC value mapped by the first target robot control instruction, so that in the process of determining the first target robot control instruction in the robot control instruction, the instruction directly mapped by the instruction semantic recognition result can be determined according to the instruction PC value mapped by the instruction semantic recognition result, and then one or more instructions are executed (such as data operation, logical operation, etc.) according to the directly mapped instruction, and the current PC value in the PC is changed to the PC value of the first target robot control instruction according to the final execution result, so that the first target robot control instruction can be determined in the voice processing instruction package.

[0049] In an embodiment, executing the first target robot control instruction to generate a robot control signal refers to an operation of obtaining a control parameter for controlling a robot according to the first target robot control instruction, and generating a robot control signal according to the robot control parameter. Wherein, obtaining the control parameter for controlling the robot according to the first target robot control instruction is various, such as that a plurality of sets of control parameters of the robot are stored in the BSRAM, the content of the first target robot control instruction can include reading one set of control parameters of the robot, so that the corresponding control parameter can be obtained by reading the control parameter according to the first target robot control instruction; again, for example, the content of the first target robot control instruction can include data operation content, and the control parameter is calculated by operation.

[0050] In an embodiment, the mechanical arm unit 2 can comprise a mechanical arm action data acquisition module 22 installed on the mechanical arm 21 and connected with the mechanical arm control SOC unit 3. The mechanical arm action data acquisition module 22 can be configured to acquire mechanical arm action data of the mechanical arm 21. The content of the first target mechanical arm control instruction can specifically comprise calculation of control parameters based on the mechanical arm action data acquired by the mechanical arm action data acquisition module 22. Specifically, in the process of executing the first target mechanical arm control instruction to generate the mechanical arm control signal, the CPU module in the mechanical arm control SOC unit 3 can first perform control data operation on the mechanical arm action data according to the first target mechanical arm control instruction to obtain first mechanical arm control data, and then generate the mechanical arm control signal according to the first mechanical arm control data. For example, the first target mechanical arm control instruction can comprise a parameter to be finally reached by the mechanical arm, and the content of the mechanical arm control data is calculated based on the difference between the parameter to be finally reached and the current mechanical arm parameter. In this way, the EXU can perform data operation based on the parameter to be finally reached and the current mechanical arm parameter based on the first target mechanical arm control instruction to determine the first mechanical arm control data, and generate the mechanical arm control signal capable of being written into the register group according to the first mechanical arm control data. In this process, since the control data operation is performed through the first target mechanical arm control instruction, the number of instruction calls in the data operation process can be reduced, thereby improving the efficiency of the generation process of the mechanical arm control signal and helping to improve the real-time performance of the mechanical arm control.

[0051] In an embodiment, the first target mechanical arm control instruction can further comprise instruction content for closed-loop adjustment of the action of the mechanical arm 21 based on data generated by the mechanical arm 21 during operation based on the mechanical arm control signal. Specifically, after the mechanical arm control signal is generated, the mechanical arm action data acquired by the mechanical arm action data acquisition module 22 during the action of the mechanical arm 21 based on the mechanical arm control signal can be acquired first, then control data operation is performed based on the mechanical arm action data to obtain second mechanical arm control data, and then the mechanical arm control signal is updated based on the second mechanical arm control data. The updating of the mechanical arm control signal based on the second mechanical arm control data can be adjustment of the action amplitude of the mechanical arm, the object clamping force, etc., which is not limited here.

[0052] For example, the first target robot arm control instruction can integrate the operation steps of a closed-loop control algorithm such as an adaptive PID algorithm and a Kalman filter algorithm in its content. When the first target robot arm control instruction is executed to generate a robot arm control signal, the robot arm action data acquisition module 22 can continuously acquire robot arm action data. Since the first target robot arm control instruction integrates a closed-loop control algorithm in its content, the second robot arm control data can be continuously calculated and the robot arm control signal can be updated according to the current robot arm action data. During this process, since the first target robot arm control instruction integrates the content of the closed-loop control algorithm, the control process continuously calculates data (i.e., the second robot arm control data) during the execution of the first target robot arm control instruction without calling other instructions, thereby achieving closed-loop control of the robot arm 21.

[0053] By continuously calculating the second robot arm control data according to the robot arm action data continuously generated during the operation of the robot arm 21, the action of the robot arm can be finely adjusted, which helps to improve the action accuracy of the robot arm. In addition, this adjustment process is performed during the execution of the first target robot arm control instruction, so that the number of instruction calls can be reduced during the entire action period of the robot arm, which helps to improve the real-time performance of the robot arm control.

[0054] In an embodiment, the robot arm action data acquisition module 22 can include at least one of an inertial measurement module and a force feedback module. The inertial measurement module can be installed on the robot arm 21 and connected to the robot arm control SOC unit 3. The inertial measurement module is used to acquire inertial data of the robot arm, wherein the inertial data can include three-axis acceleration, three-axis angular velocity, and three-axis magnetic intensity data. The force feedback module can be installed on the robot arm 21 and connected to the robot arm control SOC unit 3. The force feedback module is used to acquire force data of the robot arm. The robot arm action data can correspond to at least one of the inertial data acquired by the inertial measurement module and the force data acquired by the force feedback module.

[0055] Again referring to Figure 1 In an embodiment, the robot arm device can further include a human-computer interaction controller 4 connected to the robot arm control SOC unit. The human-computer interaction controller is used to output a target operation command according to a human-computer interaction behavior.

[0056] Referring to Figure 6 The CPU module can perform the following steps based on the human-computer interaction controller 4.

[0057] Step 210: determining a second target robot arm control instruction from the robot arm control instruction package according to the target operation command; Step 220: executing the second target robot control instruction to generate a robot control signal.

[0058] In an embodiment, the human-computer interaction behavior refers to the behavior of the user controlling the human-computer interaction controller 4 to control the robot arm 21. The specific form of the human-computer interaction controller 4 can be various, such as a remote controller, a handle, a touchable display screen, etc., which is not limited here. In addition, the human-computer interaction controller 4 can be connected to the robot control SOC unit 3 in a wireless connection manner (such as Bluetooth, WiFi, etc.), or can be connected to the robot control SOC unit 3 in a wired connection manner (such as Figure 6 In the embodiment, the human-computer interaction controller 4 is connected through a GPIO interface), which is not limited here.

[0059] In an embodiment, determining the second target robot control instruction from the robot control instruction package according to the target operation command refers to the operation of determining the instruction PC value mapped by the target operation command, determining the instruction according to the mapped instruction PC value, and determining the second target robot control instruction in the robot control instruction package. The target operation command refers to the mechanical instruction issued by the user to the robot control SOC unit 3 through the human-computer interaction controller 4 and used to control the robot arm 21 to perform the action required by the user. In addition, the instruction PC value mapped by the target operation command can be the same as or different from the instruction PC value mapped by the instruction semantic recognition result in step 110, which is not limited here. In addition, the instruction PC value determined by the target operation command can directly or indirectly map the second target robot control instruction. For example, a PC value is set for the target operation command, direct mapping refers to the case that the PC value is set to be directly mapped to the target operation command, and indirect mapping refers to the case that the PC value is different from the PC value mapped by the first target robot control instruction, but the first target robot control instruction can be finally executed by executing the instruction corresponding to the PC value.

[0060] In an embodiment, executing the second target robot control instruction to generate a robot control signal refers to the operation of obtaining a control parameter for controlling the robot arm according to the second target robot control instruction, and generating a robot control signal according to the robot control parameter. The control parameter for controlling the robot arm can be obtained in various ways according to the second target robot control instruction, such as that a plurality of sets of control parameters of the robot arm are stored in the BSRAM, the content of the second target robot control instruction can include reading one set of control parameters of the robot arm, so that the corresponding control parameter can be obtained by reading the control parameter according to the second target robot control instruction; or the content of the second target robot control instruction can include data operation content, and the control parameter is calculated by operation.

[0061] In an embodiment, in the process of executing the second target robot control instruction to generate the robot control signal, the CPU module in the robot control SOC unit 3 can first control data operation on the robot motion data collected by the robot motion data collection module 22 according to the second target robot control instruction to obtain third robot control data, and then generate the robot control signal according to the third robot control data. After generating the robot control signal, the robot motion data collected by the robot motion data collection module 22 in the process of the robot 21 moving according to the robot control signal is obtained, then control data operation is performed according to the robot motion data, fourth robot control data is obtained, and the robot control signal is updated according to the fourth robot control data. Wherein, updating the robot control signal according to the fourth robot control data can be adjusting the motion amplitude of the robot, the object clamping force, etc., which is not limited here.

[0062] By setting the human-computer interaction controller 4, the user can control the robot in ways other than voice control, thereby improving the interaction ability of the robot device and helping to improve the user's experience.

[0063] Again refer to Figure 1 In an embodiment, the robot device can also include an image collection unit 5, which is connected with the robot control SOC unit 3. The image collection unit 5 can be used to collect the environment image of the robot 21.

[0064] Refer to Figure 7 The CPU module can also be used to perform the following steps.

[0065] Step 310: In the case that the semantic content of the target voice command contains application image content, controlling the image calling environment image according to the target instruction semantic recognition result; Step 320: Identifying the environment image, and determining the first target robot control instruction from the robot control instruction package according to the obtained image recognition result.

[0066] In an embodiment, the operation of calling the environment image according to the target instruction semantic recognition result refers to determining the instruction PC value mapped by the target instruction semantic recognition result, and calling the instruction from the Reg to obtain the image collected by the image collection unit 5 and fed back. The image collection unit 5 can continuously collect images and output the collected images to the Reg for calling. In addition, the instruction for calling the environment image is various. For example, all the instructions for calling the environment image can be general instructions. In this way, when the image collection unit needs to be controlled, a plurality of general instructions can be called to perform operations to call the environment image. For another example, the instruction for controlling the image collection unit can be an instruction in the image collection instruction set constructed based on the RiscV instruction set extension. In this way, when the environment image needs to be called, the instruction in the image collection instruction set can be called to directly call the environment image from the Reg, thereby reducing the number of instructions and improving the response efficiency of the image collection unit 5.

[0067] In an embodiment, the operation of identifying the environment image refers to calling the image recognition model locally deployed in the robot arm control SOC unit 3 to identify the target in the target instruction semantic recognition result in the environment image. The image recognition model can be MobileNetV2, ShuffleNetV2, etc., which are not limited here.

[0068] Specifically, when the environment image collected by the image collection unit 5 is returned, the instruction PC value in the PC will change. The IFU can obtain the related instructions capable of calling the image recognition model from the Icache according to the instruction PC value, and then perform feature extraction, pooling, etc. on the environment image through these instructions to obtain the image recognition result.

[0069] In an embodiment, the operation of determining the first target robot arm control instruction from the robot arm control instruction package according to the obtained image recognition result refers to determining the instruction PC value mapped by the image recognition result, determining the instruction according to the mapped instruction PC value, and determining the first target robot arm control instruction in the robot arm control instruction package. The image recognition result can directly or indirectly map the first target robot arm control instruction. For example, a PC value is set for a specific image recognition result. Direct mapping refers to the case where the PC value is set to directly map the image recognition result. Indirect mapping refers to the case where the PC value is different from the PC value mapped by the first target robot arm control instruction, but the first target robot arm control instruction can be finally executed by executing the instruction corresponding to the PC value.

[0070] By setting the image acquisition unit, the mechanical arm device can have the ability to grasp objects based on image recognition, thereby improving the adaptability of the mechanical arm device to different grasping needs, and helping to improve the adaptability of the mechanical arm device to different application scenarios.

[0071] In an embodiment, in the case that the mechanical arm device can further include the image acquisition unit 5, when the human-computer interaction controller 4 is an interactive device capable of displaying the environment image (such as a touch screen, a terminal of remote control, etc.), the CPU module in the mechanical arm control SOC unit 3 can determine the second target mechanical arm control instruction based on the target operation command containing the content of the application image in the process of determining the second target mechanical arm control instruction from the mechanical arm control instruction package according to the target operation command. Specifically, in the case that the target operation command contains the content of the application image, the second target mechanical arm control instruction is determined from the mechanical arm control instruction package according to the target operation command and the environment image. For example, assuming that the human-computer interaction behavior indicates a position in the environment image, the target operation command is to clamp the object in the indicated position, at this time, the instruction PC value in the PC changes, and the CPU module calls the corresponding instruction according to the instruction PC value to identify the indicated position in the environment image, and determines the second target mechanical arm control instruction from the mechanical arm control instruction package according to the instruction PC value mapped by the identification result.

[0072] Referring to Figure 8 In an embodiment, the CPU module can further perform the following steps.

[0073] Step 410: In the case that any two of the target instruction semantic recognition result, the target operation command and the environment image wait for the processing of the other, determining the target processing order between the target instruction semantic recognition result, the target operation command and the environment image; Step 420: processing the target instruction semantic recognition result, the target operation command and the environment image according to the target processing order, and determining the third target mechanical arm control instruction from the mechanical arm control instruction package according to the processing result; Step 430: executing the third target mechanical arm control instruction to generate the mechanical arm control signal.

[0074] In an embodiment, the target instruction semantic recognition result, the target operation command and the environment image all waiting for processing means that the interval between the three is short, and the last two have completed the input operation before the first is processed. For example, the target instruction semantic recognition result, the target operation command are input in sequence in the process of calling the environment image of the previous instruction; for another example, a target operation command contrary to a target instruction semantic result requiring image recognition is input before the environment image in the process of executing the target instruction semantic result, and the like, which is not limited here.

[0075] In an embodiment, in the process of determining the target processing sequence between the target instruction semantic recognition result, the target operation command and the environment image, the processing priority between the target instruction semantic recognition result, the target operation command and the environment image can be determined first, and then the target processing sequence is determined according to the processing priority. There are various ways to determine the processing priority, such as a cloud computing based way (such as uploading the three, and determining the priority through a cloud model), a local scheduling based way (such as determining the processing priority through nested interrupts, time slice rotation, etc.), and the like. Here, no specific limitation is made.

[0076] In an embodiment, according to the target processing sequence, the target instruction semantic recognition result, the target operation command and the environment image are processed, and the third target robot arm control instruction is determined from the robot arm control instruction package according to the processing result. It means that the first one of the three is determined according to the target processing sequence, then the instruction is called according to the instruction PC value corresponding to the first one, then the instruction is called according to the instruction PC value corresponding to the second one, and finally the instruction is called according to the instruction PC value corresponding to the one with the lowest priority. It should be noted that in some cases, the processing of the second one with the second priority can be performed and completed before the processing of the third one with the third priority. For example, the instruction semantic recognition result (relatively first) is to clamp the color object, and the target operation command (relatively second) is input before the environment image (relatively third). At this time, the instruction semantic recognition result is processed first, the environment image is processed in the processing process, the instruction semantic recognition result is processed according to the processing result of the environment image, and finally the target operation command is processed.

[0077] In an embodiment, the third target robot arm control instruction is determined from the robot arm control instruction package according to the processing result, which means that the instruction PC value mapped by the processing result is determined, and the instruction is determined according to the mapped instruction PC value to determine the operation of the third target robot arm control instruction in the robot arm control instruction package. Among them, the processing result determines the mapped instruction PC value, which can directly or indirectly map the third target robot arm control instruction. For example, a PC value is set for a specific processing result. Direct mapping means that this PC value is set to directly map this image recognition result; indirect mapping means that this PC value is different from the PC value mapped by the third target robot arm control instruction, but the third target robot arm control instruction can be finally executed by executing the instruction corresponding to this PC value.

[0078] In an embodiment, the executing the third target robot control instruction to generate the robot control signal refers to an operation of obtaining a control parameter for controlling the robot according to the third target robot control instruction, and generating the robot control signal according to the robot control parameter. Wherein, the obtaining the robot control parameter according to the third target robot control instruction is various, for example, the BSRAM stores multiple sets of robot control parameters, the content of the third target robot control instruction can include reading one set of robot control parameters, so the corresponding control parameter can be obtained by reading the control parameter according to the third target robot control instruction; again, for example, the content of the third target robot control instruction can include data operation content, and the control parameter is calculated by operation.

[0079] The generation of the robot control signal is realized by determining the priority of the three, so that the robot device can support the cooperative control of multiple different control modes, thereby enabling the robot device to realize multi-modal input fusion. In addition, based on the priority of the robot control signal generation, the correct processing order can be determined in the case of delay between each modal input, thereby reducing the conflict between each modal input, and helping to improve the correctness and real-time performance of the robot 21 action.

[0080] In an embodiment, the robot device can further include a display unit 6 connected with the robot control SOC unit 3, and the display unit 6 is used to display the action of the robot 21 according to the robot control signal. Wherein, the display unit 6 can be an LCD display screen, an OLED display screen, etc., which is not limited here. In addition, the display of the robot 21 according to the robot control signal can include but not limited to displaying whether the robot 21 completes the action required by the control instruction, displaying the robot action data of the robot 21 in the process of acting according to the robot control signal, etc., which is not limited here.

[0081] By displaying the action of the robot 21 according to the robot control signal through the display unit 6, the user can be provided with intuitive visual feedback, thereby improving the interaction between the robot device and the user. In addition, since the display unit 6 can display the action of the robot 21 according to the robot control signal, the user can better issue instructions for controlling the robot, which helps to improve the robot operation ability.

[0082] In an embodiment, the mechanical arm device can further comprise a state prompting unit 7 connected with the mechanical arm control SOC unit 3, the state prompting unit 7 can be used to prompt the running state of the mechanical arm control SOC unit 3. Wherein, the state prompting unit 7 can be an LED, a sound alarm hardware (such as a buzzer), etc., which is not limited specifically here. In addition, the running state can include normal running, abnormal running, etc., which is not limited specifically here. By setting the state prompting unit 7 to prompt the current running state of the mechanical arm control SOC unit 3 to the user, the user can better control the mechanical arm 21 according to the running state, which helps to improve the user's operation experience of the mechanical arm device.

[0083] Another aspect of the embodiments of the present application also provides a robot, which can comprise the mechanical arm device as described in the above embodiments, and the implementation principle is the same as the above embodiments, which is not repeated here.

[0084] The system architecture and application scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of system architecture and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0085] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0086] In hardware implementations, the division of functionality between the functional modules / units referred to in the above description does not necessarily correspond to a division of physical components; for example, one physical component can have multiple functionalities, or one functionality or step can be performed by several physical components in cooperation. Certain physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can comprise computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0087] As used in this description, the terms "component," "module," "system," and the like are intended to refer to a computer-related entity, either hardware, software, a combination of hardware and software, software in execution, or a software component. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process or thread of execution and a component can be localized, either in whole or in part, in a single computer or distributed among multiple computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The components can communicate by way of local or remote processes such as in accordance with a signal having one or more data packets (e.g., data from components that are to interact with a local system, a distributed system, or the Internet such as according to a signal between such components) and can be stored in one or more of the data structures.

Claims

1. A robotic arm device, characterized in that: include: A speech recognition unit is used to collect speech commands, perform semantic recognition on the speech commands, and output command semantic recognition results; A robotic arm unit, comprising a robotic arm, wherein the robotic arm unit is configured to control the movement of the robotic arm according to a robotic arm control signal; A robotic arm control SOC unit is connected to the speech recognition unit and the robotic arm unit, respectively. The robotic arm control SOC unit integrates a CPU module designed based on the RiscV instruction set. The CPU module stores pre-built speech processing instruction packets and robotic arm control instruction packets. The CPU module is used to perform the following steps through a five-stage pipeline architecture: Controlling the speech recognition unit to collect a target speech command through the speech processing instruction packet, so as to obtain a semantic recognition result of the target command from the speech recognition unit; Determining a first target manipulator arm control instruction from the manipulator arm control instruction packet according to the target instruction semantic recognition result; The first target robot arm control instruction is executed to generate a robot arm control signal.

2. The robotic arm device according to claim 1, wherein: The robotic arm device also includes: A human-computer interaction controller is connected to the robot arm control SOC unit, and is used to output target operation commands according to human-computer interaction behavior; The CPU module is also used for: Determine a second target manipulator arm control instruction from the manipulator arm control instruction packet according to the target operation command; The second target robot arm control instruction is executed to generate the robot arm control signal.

3. The robotic arm device according to claim 2, wherein: The robotic arm device also includes: An image acquisition unit, connected to the robotic arm control SOC unit, and configured to acquire an environmental image of the robotic arm; The determining a first target manipulator control instruction from the manipulator control instruction packet according to the target instruction semantic recognition result includes: In the case where the semantic content of the target voice command includes the content of the application image, calling the environment image according to the semantic recognition result of the target command; The environmental image is recognized, and the first target manipulator arm control instruction is determined from the manipulator arm control instruction packet according to the obtained image recognition result.

4. The robotic arm device according to claim 3, wherein: The determining a second target manipulator arm control instruction from the manipulator arm control instruction packet according to the target operation command includes: In a case where the target operation command includes content of an application image, the second target robotic arm control instruction is determined from the robotic arm control instruction packet according to the target operation command and the environment image.

5. The robotic arm device according to claim 4, characterized in that: The CPU module is also used for: determining a target processing order among the instruction semantic recognition result, the target operation command, and the environment image when any two of the instruction semantic recognition result, the target operation command, and the environment image are waiting for processing by the other; Processing the instruction semantic recognition result, the target operation command, and the environment image according to the target processing order, and determining a third target manipulator arm control instruction from the manipulator arm control instruction packet according to the processing result; The third target robot arm control instruction is executed to generate the robot arm control signal.

6. The robotic arm device according to claim 1, wherein: The robotic arm unit comprises: A manipulator motion data acquisition module is installed on the manipulator and connected to the manipulator control SOC unit, and is used to collect manipulator motion data of the manipulator; The executing the first target robotic arm control instruction to generate a robotic arm control signal includes: performing a control data operation on the robotic arm motion data according to the first target robotic arm control instruction to obtain first robotic arm control data; The robot arm control signal is generated according to the first robot arm control data.

7. The robotic arm device according to claim 6, wherein: The CPU module is also used for: After generating the robotic arm control signal, acquiring the robotic arm motion data collected by the robotic arm motion data collection module during the process of the robotic arm performing motion according to the robotic arm control signal; Performing control data calculation according to the robot arm motion data to obtain second robot arm control data; The robot arm control signal is updated according to the second robot arm control data.

8. The robotic arm device according to claim 7, wherein: The robotic arm motion data acquisition module includes at least one of the following: an inertial measurement module, mounted on the robotic arm and connected to the robotic arm control SOC unit, the inertial measurement module being used to collect inertial data of the robotic arm; A force feedback module is installed on the robotic arm and connected to the robotic arm control SOC unit, and is used to collect force data of the robotic arm; The robot arm motion data includes at least one of the following: The inertial data collected by the inertial measurement module; The force data collected by the force feedback module.

9. The robotic arm device according to claim 5, characterized in that: The robotic arm device also includes: A display unit is connected to the robot arm control SOC unit, and the display unit is used to display the situation in which the robot arm moves according to the robot arm control signal.

10. A robot, characterized in that: Comprising the robotic arm device according to any one of claims 1 to 9.

Images