RISC-V-based attitude recognition method and system

Abstract

The invention discloses a posture recognition method and system based on RISC-V. The method comprises the following steps: acquiring a video stream image of a target object; preprocessing the acquired video stream image, inputting the preprocessed video stream image into a pre-constructed LightPose V2 attitude recognition network model, and performing attitude recognition by adopting a task division strategy; the light weight processing is carried out on the pre-constructed LightPose V2 attitude recognition network model by using an integer algorithm and quantitative perception training; the posture recognition process is completed in an independent algorithm accelerator. According to the method, end-to-end acceleration of real-time posture recognition in an Internet of Things scene is achieved, lightweight design is completed, the purposes of low power consumption and low delay are achieved, the accuracy and real-time performance of posture recognition are greatly improved, omnibearing optimization from a single link to the whole system is achieved, and the method is suitable for popularization and application. And more reliable and efficient technical support is provided for applications such as robot control.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision recognition based on artificial intelligence, and relates to a posture recognition method and system based on RISC-V. Background Art

[0002] With the rapid development of Internet of Things technology and edge computing systems, posture recognition, as an important technical means, has been widely used in many fields. For example, fields such as autonomous driving, intelligent monitoring, and motion recognition all rely on high-precision posture recognition technology to achieve their functions. In recent years, with the rapid development of deep learning, various convolutional neural network models such as convolutional posture machines, OpenPose, and SimpleBaseline have been proposed one after another, and have made significant progress in posture recognition tasks.

[0003] In practical applications, IoT devices usually have limited computing resources and storage space, which makes it difficult to support the operation of large-scale deep learning models. In addition, the power consumption and heat dissipation capabilities of these devices are also limited, further limiting the application of complex models. Although algorithms such as CPM, OpenPose, and SimpleBaseline perform well in laboratory environments, their deployment in real scenarios faces many challenges. For example, autonomous driving systems need to process a large amount of sensor data and make accurate posture recognition in a very short time to ensure the safe driving of vehicles; intelligent monitoring systems need to analyze the human movements in video streams in real time to detect abnormal behavior and issue timely alarms. These applications place extremely high demands on the inference speed and accuracy of posture recognition.

[0004] To solve these problems, researchers have proposed many lightweight models and optimization strategies. Lightweight models reduce resource consumption by reducing model parameters and computational complexity while maintaining the accuracy of the model as much as possible. Some studies focus on model pruning and compression, knowledge distillation, and lightweight network structure design. For example, pruning technology can remove unimportant parameters in the model, thereby reducing the size and computational complexity of the model; knowledge distillation improves the performance of the lightweight model by transferring the knowledge of the complex model to the lightweight model. In addition, designing a specialized lightweight network structure is also an important way to improve model efficiency.

[0005] The above methods have indeed improved the efficiency of the model accordingly, but in the posture recognition task, the complexity of the system requires careful consideration and optimization in multiple aspects, rather than simply considering improving efficiency through lightweight models. Specifically, the posture recognition task involves not only image acquisition and image preprocessing, but also posture recognition acceleration and robot control output. However, current research mainly focuses on lightweight strategies for some key technologies, and pays insufficient attention to the overall system-level software and hardware collaborative optimization. Although this one-sided technical optimization can improve the efficiency of a single link, it fails to fully explore the potential of the system from a global perspective, resulting in the overall operating efficiency failing to reach the optimal state. Summary of the invention

[0006] In view of the problems existing in the prior art, the present invention provides a posture recognition method and system based on RISC-V, so as to solve the technical problems in the prior art that the system model has poor overall operating efficiency, high power consumption and long time consumption when performing posture recognition tasks.

[0007] The present invention is achieved through the following technical solutions: A gesture recognition method based on RISC-V, comprising the following steps: S1: Acquire the video stream image of the target object; S2: Preprocess the acquired video stream images, input the preprocessed video stream images into the pre-built LightPose V2 posture recognition network model, and use the task division strategy for posture recognition; The pre-built LightPose V2 posture recognition network model is lightweight using integer arithmetic and quantization-aware training; the posture recognition process is completed in an independent algorithm accelerator.

[0008] Preferably, the preprocessing includes denoising, contrast enhancement and edge detection.

[0009] Preferably, the task division strategy is used for gesture recognition, specifically: If the image acquisition is not completed and the recognition of the previous image has been completed, stop the image acquisition, select the output data, and continue the unfinished image acquisition after the data output is completed; If the image acquisition is completed, it is detected whether there is an idle algorithm accelerator. If there is an idle algorithm accelerator, the acquired image is transmitted to the idle algorithm accelerator for gesture recognition. Otherwise, gesture recognition is performed after the algorithm accelerator completes its work.

[0010] Preferably, the pre-built LightPose V2 posture recognition network model includes a feature extraction network and an upsampling layer.

[0011] Preferably, the algorithm accelerator includes a Block1 module, a Block2 module and a Block3 module; The Block1 module includes a convolution layer, a depth convolution and a point convolution, and implements a preprocessing process; The Block2 module includes a padding and sliding window unit module, a matrix vector unit module and a post-processing module, which are used for residual learning of the residual layer; The Block3 module includes a point-by-point convolution module and a depth-wise separable transposed convolution module, which are used for feature upsampling and heat map generation.

[0012] Preferably, the quantization process includes activation quantization and weight quantization; layer-by-layer quantization is used for the activation, and channel-by-channel quantization is used for the weight.

[0013] A gesture recognition system based on RISC-V, comprising: Data acquisition unit: The data acquisition unit is used to acquire a video stream image of a target object; Posture recognition unit: The posture recognition unit is used to preprocess the acquired video stream images, input the preprocessed video stream images into a pre-built LightPose V2 posture recognition network model, and adopt a task division strategy for posture recognition; the pre-built LightPose V2 posture recognition network model is lightweight processed using integer algorithms and quantization-aware training; the posture recognition process is completed in an independent algorithm accelerator.

[0014] A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0015] A computer-readable storage medium stores a computer program, wherein the computer program / instructions are executed by a processor to implement the steps of the above method.

[0016] A computer program product comprises a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0017] Compared with the prior art, the present invention has the following beneficial technical effects: The invention discloses a posture recognition method based on RISC-V. The LightPose V2 posture recognition network model used in the method performs posture recognition. The network model performs lightweight processing using integer algorithms and quantization perception training. The lightweight processing process not only significantly reduces the amount of calculation and parameter requirements, but also controls the precision loss within a negligible range, reducing the calculation complexity and network size by more than 50 times. In addition, the posture recognition process is completed in an independent algorithm accelerator, and in the recognition process, a task division strategy is adopted to complete the posture recognition process. Here, the posture recognition process is completed in an independent algorithm accelerator, and can be performed in parallel with tasks such as image acquisition, image preprocessing, robot control output, and HDMI display. At the same time, the task division strategy completes the posture recognition process, that is, after task division, task scheduling and pipeline processing, efficient pipeline parallel processing of the entire system is realized. The method not only realizes end-to-end acceleration of real-time posture recognition in an Internet of Things scenario, completes lightweight design, and achieves the purpose of low power consumption and low latency, but also greatly improves the accuracy and real-time performance of posture recognition, realizes all-round optimization from a single link to the entire system, and provides more reliable and efficient technical support for applications such as robot control. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is the image recognition flow chart of the present invention; Figure 2 It is the architectural design diagram of the gesture recognition heterogeneous computing system (System on Chip, SoC) in the present invention; Figure 3 It is the LightPose V2 network structure diagram in the present invention; Figure 4 It is Algorithm 1 in the present invention; Figure 5 This is Algorithm 2 in the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0023] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0024] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0025] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0026] The present invention is further described in detail below in conjunction with the accompanying drawings: Example 1 The present invention discloses a posture recognition method based on RISC-V, comprising the following steps: S1: Acquire the video stream image of the target object; S2: preprocessing the acquired video stream image, wherein the preprocessing includes cropping and resizing, inputting the preprocessed video stream image into a pre-built LightPose V2 posture recognition network model, and performing posture recognition using a task division strategy; The pre-built LightPose V2 posture recognition network model is light-weighted using integer arithmetic and quantization-aware training. The quantization process includes activation quantization and weight quantization. The activation is quantized layer by layer, and the weight is quantized channel by channel.

[0027] In addition, the posture recognition process is completed in an independent algorithm accelerator, while tasks such as image acquisition, image preprocessing, robot control output, and HDMI display are completed on the RISC-V CPU side.

[0028] The above-mentioned task division strategy is used for posture recognition, specifically: If the image acquisition is not completed and the recognition of the previous image has been completed, stop the image acquisition, select the output data, and continue the unfinished image acquisition after the data output is completed; If the image acquisition is completed, it is detected whether there is an idle algorithm accelerator. If there is an idle algorithm accelerator, the acquired image is transmitted to the idle algorithm accelerator for gesture recognition. Otherwise, gesture recognition is performed after the algorithm accelerator completes its work.

[0029] The above pre-built LightPose V2 pose recognition network model includes a feature extraction network and an upsampling layer; The algorithm accelerator includes a Block1 module, a Block2 module and a Block3 module; The Block1 module includes a convolution layer (Conv-convolution, Conv), a depth convolution (DCBR-Depth-wise Conv, BN, ReLU, DCBR) and a point convolution (PCB-Point-wise Conv, BN, PCB), and implements a preprocessing process; The Block2 module includes a padding and sliding window unit module, a matrix vector unit module and a post-processing module, which are used for residual learning of the residual bottleneck block (RBB); The Block3 module includes a point-by-point convolution module and a depth-wise separable transposed convolution module, which are used for feature upsampling and heat map generation.

[0030] The post-processing module includes a bias adding submodule, a quantization submodule, a residual adding submodule and a ReLU submodule.

[0031] That is, the present invention also includes a SoC design method, include: Set up the CPU, i.e. RISC-V CPU, and algorithm accelerator; A data processing mechanism is set up, the CPU executes the main program, and the algorithm accelerator executes the LightPose V2 posture recognition network model algorithm.

[0032] Specifically, the present invention proposes a RISC-V-based lightweight gesture recognition heterogeneous computing system (SoC) design to achieve end-to-end acceleration of real-time gesture recognition in IoT scenarios, complete lightweight design, and achieve low power consumption and low latency. Its technical solution is: On the algorithm side, the posture recognition network was optimized based on Microsoft's Simple Baseline posture recognition network to obtain the LightPose V2 posture recognition network. In terms of quantization, the method of using only integer arithmetic (IAO) was applied, combined with quantization-aware training, and in order to minimize element operations, additional shortcut optimization was selected in the data quantization process. The model parameters were reduced by quantization and compression, and the overall calculation amount was compressed. After calculation, the model successfully reduced the computational complexity and network size by more than 50 times. The purpose of lightweighting was achieved.

[0033] The hardware side designs a dedicated algorithm accelerator for the core structure of the posture recognition network model, and splits the computing modules according to the network structure. Since the input image has only three channels (RGB), the first three layers are assigned to a separate module called Block1 together with the preprocessing to improve parallelism. In order to obtain higher performance, the upsampling layer is implemented in Block3, including Pwcv and depth-separable transposed convolution. Block2 is used for RBB to reduce the required DDR bandwidth. The computing engine for RBB includes padding, SWU, MVU, and post-processing modules, and schedules each hardware module in a module-level pipeline manner to improve the system computing efficiency.

[0034] like Figure 1 As shown in the figure, the overall task scheduling strategy of the system is optimized, and an efficient task division strategy is adopted to dispatch tasks such as image acquisition, image preprocessing, robot control output, and HDMI display to the CPU side, and the core operation of the posture recognition network is dispatched to the algorithm accelerator. Efficient pipeline parallel processing of the entire system is achieved. The specific operations are as follows: Step 1: Receive the corresponding task release signal and start image acquisition of the task target Step 2: Image preprocessing after image acquisition Step 3: Transfer the preprocessed image to the CNN algorithm accelerator for posture recognition Step 4: Repeat steps 1 and 2. If the information extraction of the previous image is completed before steps 1 and 2 are completed, stop steps 1 and 2 first, select output data, and then proceed to the unfinished steps 1 and 2 after the data output is completed. If steps 1 and 2 have been completed, perform a test, and if there is an idle algorithm accelerator, transfer it to the algorithm accelerator for gesture recognition. If there is no idle algorithm accelerator, wait for the algorithm accelerator to complete its work before performing gesture recognition.

[0035] Step 5: Repeat the above four steps until the work is completed.

[0036] The entire SoC design contains a RISC-V CPU for system control, an ACC module for accelerating CNN calculations, and several peripherals for data storage and transmission. The RISC-V CPU is responsible for configuring the network structure information and memory space of the accelerator. At runtime, the captured video stream is preprocessed (cropped and resized) before being written to DDR. The accelerator performs network inference to obtain a heat map of the human posture, and then obtains the joint coordinates from the heat map. The display module reads the data stream from DDR for post-processing (resize and labeling) and completes the display. The system operation status is monitored and controlled by the RISC-VCPU and can be accessed through the UART link.

[0037] Through the optimization of software and hardware, collaborative work, and finally the optimization of the entire task scheduling strategy, an integrated high-performance, low-power, low-latency RISC-V-based lightweight gesture recognition heterogeneous computing system (SoC) was built.

[0038] This paper presents a new design of RISC-V SoC based on FPGA, which is specially designed for end-to-end real-time gesture recognition. To achieve this goal, a lightweight convolutional neural network (CNN) is proposed, and the integer-only arithmetic (IAO) method is applied, combined with quantization-aware training. Throughout the training process, feature maps and network weights are quantized to 8-bit integers, using layer-by-layer quantization for activations and channel-by-channel quantization for weights. This quantization not only significantly reduces the amount of computation and parameter requirements, but also keeps the accuracy loss within a negligible range. The accelerator is optimized with a full dataflow computing engine and pipeline modules to further improve processing efficiency.

[0039] In SoC systems, task division, task scheduling, and pipeline processing between the CPU and the acceleration module are key factors affecting overall performance. To this end, an efficient task division mechanism is designed to reasonably allocate different types of tasks to the CPU and the acceleration module to give full play to the parallel computing capabilities of the hardware. For example, for computationally intensive image processing tasks, they can be handled by customized acceleration modules, while for tasks with complex control logic, the CPU is responsible for execution. In addition, an intelligent task scheduling algorithm is used to dynamically adjust the execution order of tasks according to the priority and resource requirements of the tasks, avoid resource conflicts and idleness, and improve the response speed and processing power of the system.

[0040] Through this all-round optimization from algorithm to hardware, from a single link to the entire system, efficient algorithm and hardware coordination is achieved, significantly improving the overall operating efficiency of the gesture recognition task. Experimental results show that the present invention not only effectively reduces the power consumption and latency of the system, but also greatly improves the accuracy and real-time performance of gesture recognition, providing more reliable and efficient technical support for applications such as robot control. These optimization measures not only improve the performance of the system, but also expand its scope of application, enabling it to play a role in more complex and demanding scenarios.

[0041] Example 2 In order to further explain the technical solution of the present invention, the following embodiments are used for illustration: like Figure 2 As shown in the figure, the entire SoC design includes a RISC-V CPU for system control, an ACC module for accelerating CNN calculations, and several peripherals for data storage and transmission.

[0042] First, in terms of algorithms: Based on Microsoft's Simple Baseline posture recognition network, the posture recognition network was optimized to obtain the LightPose V2 posture recognition network. The specific operations are as follows: (1) In the feature extraction stage, the MobileNetV2 network (width multiplier = 0.5, that is, the number of output channels with a channel multiplication factor of 0.5, and when the number of channels is not a multiple of 8, it will be expanded to a multiple of 8) is selected to replace the original ResNet in the feature extraction stage.

[0043] (2) In the upsampling stage, a scheme is adopted to decompose the ordinary transposed convolution into one point-wise convolution (Pwcv) operation and one depth-wise separable transposed convolution (Tconv) operation for feature upsampling and heat map generation.

[0044] Figure 3The main structure of the developed lightweight pose recognition model LightPose V2 is shown. Each convolutional layer is followed by a ReLU layer and a BN layer.

[0045] Quantization is usually performed after model building. When quantization is applied to lightweight models (such as LightPose V2) during quantized training, it usually leads to a significant drop in inference accuracy. To address this issue, an approach using only integer arithmetic (IAO) is applied, combined with quantization-aware training. Throughout the training process, feature maps and network weights are quantized to 8-bit integers, using layer-by-layer quantization for activations and channel-by-channel quantization for weights. To further reduce the network size and computational complexity, the BN layer is fused into the layer before it. Basic operations after convolution, such as adding biases and shortcuts, may increase the length of the critical path, resulting in reduced processing speed and increased power consumption for hardware implementations. In order to minimize element-wise operations, additional shortcut optimizations are applied during data quantization. The shortcut operations in the integer inference stage can be expressed as:

[0046] Where q represents the quantized data and S represents the floating point scale value. Then the quantized output of the shortcut can be expressed as:

[0047] During quantization-aware training, if we , then the output of the shortcut connection can be directly calculated by the adder without any loss of precision.

[0048] Secondly, in terms of hardware: the hardware architecture of the algorithm accelerator is as follows: Figure 2 As shown in Figure 1. In order to fully pipeline the design, the computational modules are split according to the network structure. Since the input image has only three channels (RGB), the first three layers are assigned to a separate module called Block1 along with the preprocessing to improve parallelism. In order to achieve higher performance, the upsampling layer is implemented in Block3, including Pwcv and depth-separable transposed convolution. Block2 is used for RBB to reduce the required DDR bandwidth. The three modules work in parallel in a pipelined manner. For each module, it reads FM data from its previous module and writes the output data back to the off-chip memory. All three pipeline blocks read data from one DDR memory bank for processing and write the processed data back to another memory bank. The computational engine for RBB contains padding, SWU, MVU, and post-processing modules. Since there is no need to buffer inputs and intermediate results in the design, the use of on-chip BRAM can be saved.

[0049] Padding and Sliding Window Unit (SWU): The padding module fills the input data stream with zero values, including padding zero values ​​in Dwcv and inserting zero values ​​between input data in Tconv. SWU uses a full data flow method to expand the convolution into a data flow of matrix-vector operations.

[0050] Matrix Vector Unit (MVU): MVU consists of PE, multiplier and accumulator. In order to make full use of the multiplication resources on the FPGA, each Xilinx DSP48E1 is divided into two 8-bit × 8-bit multipliers so that the two multipliers can share one input port. The specific multiplexing method is as follows: Assuming that a, b, and c are three signed 8-bit data, a is first shifted left by 16 bits and expanded into a 25-bit signed number tempa as the input data of the DSP unit input port A; b is extended to become a 25-bit signed number tempb as the input data of the DSP unit input port D; c is expanded into an 18-nit signed number tempc. As the input data of the DSP unit input port B. The pre-addition unit in the DSP is used to calculate the sum of tmpa and tmpb, and then the 25-bit sum and the 18-bit tempc are sent to the multiplier in the DSP unit. Then, the calculation results of the two-way multiplication are intercepted from the 43-bit multiplication result P of the DSP, where the lower 16 bits of P are the product of b and c, and the lower 31-16 bits of P plus the sign of bc are P

[15] , which is the product of a and c. In this way, the deployment of two-way 8-bit multiplication on one DSP unit is completed. Using the DSP multiplexing method, one DSP unit can complete two-way 8-bit multiplication operations at the same time, which greatly improves the utilization rate of the DSP unit. In one clock cycle, the DSP unit that could only complete one multiplication operation can now complete two operations, doubling the efficiency. Secondly, this method reduces the time delay of data processing. Since the two-way multiplication operations can be performed simultaneously, the overall calculation time is shortened, thereby improving the real-time response capability of the system. In addition, this method can also reduce the power consumption of the system. Because the shortened calculation time means that the DSP unit can enter the low-power mode earlier in the same time, thus saving energy.

[0051] For the Sconv layer, one FM channel data is shared by two weight data. For the Dwcv layer, a multiplier array (MUL) is used to complete vector-vector multiplication. The accumulator consists of some adders, which are used for channel accumulation in Pwcv and kernel accumulation in Dwcv. The specific process is as follows: The accumulation buffer (Acc Buffer) is used to temporarily save the intermediate operation results from the multiplication and addition module. When a new intermediate operation result is transmitted from the multiplication and addition module, the previous intermediate operation result is read out from the accumulation buffer and added to the output of the new multiplication and addition module. The accumulation result is sent back to the accumulation buffer to update the previous intermediate result. After the accumulation module completes the calculation and obtains the output feature map result, the calculation result is read out from the accumulation buffer and written to the output, and transmitted to the subsequent modules to complete the convolution post-processing operation.

[0052] Post-processing: The post-processing module contains add bias, quantization, add residual and ReLU sub-modules. It works selectively according to the structure of different layers.

[0053] Full data flow mode: Special optimizations for pipeline computing and full data flow strategies are used, combined with loop unrolling strategies, and full data flow mode is applied to FM to save data loading time and BRAM usage while simplifying control logic. For the Sconv layer, a SIMD input channel slice with a dimension of 1×1 and kernel data with a depth of SIMD×PE are read in each clock cycle. Multiplication and addition operations are performed in a loop until the entire kernel is traversed and a complete output FM slice with a depth of PE is obtained. A row buffer is used to store FM data with the same height as the kernel to expand the FM bandwidth. For the Dwcv layer, the feature map is first taken out in the order of the kernel (not the channel) to save the use of on-chip buffers, because these intermediate results in the channel direction do not need to be added up.

[0054] Different from the hardware implementation of the Sconv layer, which mainly focuses on loop unrolling exploration, the design of the Tconv layer requires a trade-off between hardware overhead and efficiency. In order to eliminate inefficient zero padding operations with minimal resource overhead, Tconv is decomposed to standardize the computational modes of Tconv and Dwcv. As shown in Figure 4, the zero values ​​in the depthwise separable convolution (Tconv) are inserted into the row buffer before upsampling, which only requires a small amount of expansion in the padding module. Therefore, the depthwise separable convolution (Tconv) can use the same computational module as Dwcv.

[0055] Loop unrolling: In LightPose, the sizes of intermediate layers vary widely. The size of feature maps ranges from 8×6 to 128×96, and the number of channels ranges from 8 to 160. In order to explore full adaptive computational parallelism in different convolutional layers, convolutions are unrolled from the input and output channels, as shown in Algorithm 1 (see Figure 4 As shown) and Algorithm 2 (as shown Figure 5 For Sconv, the innermost two loops have fixed PE and SIMD loop bounds and can be fully unrolled in one clock cycle. This loop unrolling method can be viewed as a matrix-vector multiplication. Pwcv can be calculated in the same way as Sconv, with K=1. Unlike Sconv, which is viewed as a vector-vector multiplication, Dwcv is unrolled in only one dimension of the input and output channels.

[0056] The present invention is an optimization design strategy for an integrated system from algorithm to hardware and finally the entire task scheduling strategy. The specific workflow of the integrated system is as follows: (1) When the system receives a task release signal, it will immediately start and enter the working state. After the resource scheduler receives the task, it detects the number of currently idle accelerators N and can start the task. Then the system triggers the image acquisition process and uses a high-resolution camera or sensor to capture clear and detailed image data. These images are the basis for subsequent processing and analysis, so it is very important to ensure their quality.

[0057] (2) After successfully acquiring the image, the system will perform a series of preprocessing operations on the acquired image. The purpose of image preprocessing is to improve the quality of the image, enhance key features, and prepare for the subsequent gesture recognition steps. Common preprocessing methods include denoising, contrast enhancement, edge detection, etc. Through these processing methods, the noise in the image can be effectively reduced, the contrast of the image can be improved, and the edges of the target object can be highlighted, making the image more suitable for further analysis.

[0058] (3) The preprocessed image will be transmitted to the convolutional neural network (CNN) accelerator for posture recognition, and the resource scheduler detects that the number of accelerators occupied is N0.

[0059] (4) The system will continue to repeat (1) and (2), that is, it will continuously perform image acquisition and preprocessing. If the information extraction of the previous image is completed before (1) and (2) are completed, the system will temporarily stop the current image acquisition and preprocessing work. In this case, the system will choose to output the extracted data first to ensure timely response and processing. Once the data output is completed, the system will return to the unfinished image acquisition and preprocessing steps and continue to complete these tasks.

[0060] If steps 1 and 2 have been completed, the following operations are performed: the resource scheduler detects whether the remaining number of accelerators N1=N-N0 is sufficient to perform gesture recognition. If so, the system will immediately transfer the preprocessed image to the accelerator for gesture recognition. If not, the system will continue to perform operations (1) and (2) until an accelerator completes the current task and releases resources.

[0061] (5) The system will continue to loop through the above four steps until all tasks are completed. This process may take a long time, depending on the number and complexity of the tasks.

[0062] Through the above task division strategy, the advantages of different hardware units can be fully utilized to achieve efficient pipeline parallel processing of the entire system. The CPU is responsible for logic control and tasks with strong serial processing capabilities, while the accelerator focuses on high-density parallel computing tasks, thereby maximizing system performance.

[0063] In addition to the above-mentioned software and hardware collaboration, the system is also equipped with an intelligent task scheduling algorithm. In complex scenarios, the system will implement task scheduling strategies for different task sizes. After extracting relevant data information, the system will first determine the task size through its own functions or monitoring modules, and then the CPU will be reasonably scheduled according to the task scheduling strategy, so that the CPU and accelerator can run efficiently and collaboratively, minimizing idle time and maximizing the overall system throughput.

[0064] After various optimizations and improvements, from algorithm improvement to hardware upgrade, to careful polishing of details and optimization of the overall architecture, an efficient integrated system has been successfully created. The system significantly improves the execution efficiency of gesture recognition tasks. Experimental data verifies the effectiveness of this innovative measure, which not only reduces the energy consumption and response time of the system, but also significantly improves the accuracy and real-time performance of gesture recognition, laying a solid technical foundation for the application of cutting-edge technologies such as robot control. These optimization measures have significantly improved the system performance and expanded its application range, enabling it to play a key role in more complex and demanding environments.

[0065] Example 3 The present invention also provides a RISC-V-based gesture recognition system, comprising: Data acquisition unit: The data acquisition unit is used to acquire a video stream image of a target object; Posture recognition unit: The posture recognition unit is used to preprocess the acquired video stream images, input the preprocessed video stream images into a pre-built LightPose V2 posture recognition network model, and adopt a task division strategy for posture recognition; the pre-built LightPose V2 posture recognition network model is lightweight processed using integer algorithms and quantization-aware training; the posture recognition process is completed in an independent algorithm accelerator.

[0066] In addition, a schematic diagram of a terminal device is provided in an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0067] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0068] The terminal device may be a computing device such as a desktop computer, a notebook, a palmtop computer, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0069] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0070] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0071] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals. The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0072] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A gesture recognition method based on RISC-V, characterized in that: The following steps are involved: S1: Acquire the video stream image of the target object; S2: Preprocess the acquired video stream images, input the preprocessed video stream images into the pre-built LightPose V2 posture recognition network model, and use the task division strategy for posture recognition; The pre-built LightPose V2 posture recognition network model is lightweight using integer arithmetic and quantization-aware training; the posture recognition process is completed in an independent algorithm accelerator.

2. A gesture recognition method based on RISC-V according to claim 1, characterized in that: The preprocessing includes denoising, contrast enhancement and edge detection.

3. The RISC-V-based gesture recognition method according to claim 1, characterized in that: The task division strategy is used for gesture recognition, specifically: If the image acquisition is not completed and the recognition of the previous image has been completed, stop the image acquisition, select the output data, and continue the unfinished image acquisition after the data output is completed; If the image acquisition is completed, it is detected whether there is an idle algorithm accelerator. If there is an idle algorithm accelerator, the acquired image is transmitted to the idle algorithm accelerator for gesture recognition. Otherwise, gesture recognition is performed after the algorithm accelerator completes its work.

4. A gesture recognition method based on RISC-V according to claim 1, characterized in that: The pre-built LightPose V2 posture recognition network model includes a feature extraction network and an upsampling layer.

5. The RISC-V-based gesture recognition method according to claim 1, characterized in that: The algorithm accelerator includes a Block1 module, a Block2 module and a Block3 module; The Block1 module includes a convolution layer, a depth convolution and a point convolution, and implements a preprocessing process; The Block2 module includes a padding and sliding window unit module, a matrix vector unit module and a post-processing module, which are used for residual learning of the residual layer; The Block3 module includes a point-by-point convolution module and a depth-wise separable transposed convolution module, which are used for feature upsampling and heat map generation.

6. The RISC-V-based gesture recognition method according to claim 1, characterized in that: The quantization process includes activation quantization and weight quantization; layer-by-layer quantization is used for the activation, and channel-by-channel quantization is used for the weight.

7. A gesture recognition system based on RISC-V, characterized in that: include: Data acquisition unit: The data acquisition unit is used to acquire a video stream image of a target object; Posture recognition unit: The posture recognition unit is used to preprocess the acquired video stream images, input the preprocessed video stream images into a pre-built LightPose V2 posture recognition network model, and adopt a task division strategy for posture recognition; the pre-built LightPose V2 posture recognition network model is lightweight processed using integer algorithms and quantization-aware training; the posture recognition process is completed in an independent algorithm accelerator.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method as described in any one of claims 1 to 6 are implemented.

Images