Device Control Method, System, Apparatus and Product Based on Multimodal Multi-Core Heterogeneous

Abstract

The present application provides a device control method, system, apparatus and product based on multi-modal multi-core heterogeneous. The method includes: obtaining data to be transmitted from a multi-modal media data acquisition module and storing it in a data buffer; a soft interrupt unit determines the backlog status of data transmission tasks in the data buffer. If there is a backlog, all the data to be transmitted in the data buffer is packed and stored in a specified storage area. If there is no backlog, the data to be transmitted is sequentially read and stored in the specified storage area; a hard interrupt unit reads the information to be transmitted from the specified storage area and sends it to a media data processing module, and the media data processing module is configured to obtain an analysis result of control instructions; based on the analysis result of control instructions, the control instructions are sent to the controlled device. The present application is applied to a multi-core heterogeneous device control system, reducing the waiting time for transmitting and processing multi-modal data and improving the real-time performance of control.

Description

Background Art

[0002] When the teacher is giving a lecture, it is necessary to control the PPT (slides made with office PowerPoint), videos, etc. displayed on the computer device (such as turning the PPT page, controlling the start of video playback, pausing video playback, etc.). The control of traditional classroom equipment relies on physical remote control or fixed touch operations, which have problems such as high response latency and poor multi-modal instruction coordination. Moreover, multiple physical interactions may be required during device control (such as remote control switching + touch screen confirmation), lacking an adaptive threshold mechanism, and users need to manually calibrate the device sensitivity and cannot adapt to different scenario differences.

[0003] To facilitate the teacher's operation and reduce the teacher's manual control of the computer device, a control system has emerged that captures the teacher's gesture commands through a camera and then controls the courseware playback process of the computer device (such as controlling the courseware page turning, controlling video playback, controlling picture zooming, etc.). In the prior art, a control system based on a single modality (visual image modality) is easily interfered by the environment (such as background noise, light changes). In dynamic lighting (such as strong light interference from a projector) or occlusion scenarios caused by chandeliers, etc., the target capture accuracy drops significantly. This control system involves the cooperation and interaction of multiple devices such as cameras, control modules, video data processing devices, and controlled devices. Affected by the network communication latency of multiple devices or the differences in split media data acquisition devices, when transmitting control instructions through a general network protocol (such as TCP / IP, Transmission Control Protocol / Internet Protocol), multiple protocol conversions are required, adding an additional delay of 20 - 50 ms. If a cloud platform is used to process video data, single-modal tasks monopolize computing resources (such as a GPU (Graphics Processing Unit) only processing images), resulting in an imbalance in CPU (Central Processing Unit) / GPU utilization, general instruction processing latency, and low hardware resource scheduling efficiency between different devices, making it difficult to meet the real-time requirements of teaching. Summary of the Invention

[0004] Aiming at the problems in the prior art, the purpose of this application is to provide a device control method, system, device, and product based on multi-modal multi-core heterogeneous, which is applied to a multi-core heterogeneous device control system, reduces the waiting time for transmitting and processing multi-modal data, and improves the real-time performance of control.

[0005] The first aspect of this application provides a device control method based on multi-modal multi-core heterogeneous, which is implemented based on a control module. The method includes the following steps:

[0006] Obtain the data to be transmitted from the multimodal media data acquisition module and store it in the data buffer;

[0007] Every first preset time, the soft interrupt unit determines the backlog status of the data transmission tasks in the data buffer. If there is a backlog, pack all the data to be transmitted in the data buffer and store it in the specified storage area. If there is no backlog, sequentially read the data to be transmitted in the data buffer and store it in the specified storage area;

[0008] Every second preset time, the hard interrupt unit reads the information to be transmitted from the specified storage area. The information to be transmitted includes the data to be transmitted or the data packet, and sends it to the media data processing module. The media data processing module and the control module have different processing cores, and the media data processing module is configured to analyze the received data or data packet to obtain the analysis result of the control instruction;

[0009] Send the control instruction to the controlled device based on the analysis result of the control instruction determined by the media data processing module.

[0010] In some embodiments, the information to be transmitted further includes a command to be transmitted, and the method further includes the following steps:

[0011] Obtain the command to be transmitted and store it in the command buffer. Among them, if the command transmission task in the command buffer is backlogged and the current number of command transmission tasks does not exceed the maximum command number threshold, expand the capacity of the command buffer. If the number of command transmission tasks in the command buffer exceeds the maximum command number threshold, do not expand the capacity and issue an exception notification;

[0012] Every first preset time, the soft interrupt unit reads the command to be transmitted from the command buffer and stores it in the specified storage area.

[0013] In some embodiments, the soft interrupt unit is further used to read the command to be transmitted from the command buffer and store it in the specified storage area, and the information to be transmitted further includes the command to be transmitted;

[0014] The steps for the hard interrupt unit to read the information to be transmitted from the specified storage area include:

[0015] The hard interrupt unit determines the priority of the information to be transmitted in the specified storage area;

[0016] The hard interrupt unit sequentially reads the information to be transmitted in the specified storage area according to the priority order and sends it to the media data processing module; among them, the hard interrupt unit reads the information to be transmitted with a priority higher than the preset level threshold and sequentially sends it to the media data processing module, and packs the information to be transmitted with a priority lower than or equal to the preset level threshold and sends it to the media data processing module.

[0017] In some embodiments, a common error code lookup table for the control module and the media data processing module is pre-constructed, and a policy lookup table corresponding to the common error code and the error handling policy is pre-constructed;

[0018] The method further includes the following steps:

[0019] Receive exception information from the control module or the media data processing module;

[0020] Based on the common error code lookup table, determine the common error code corresponding to the exception information;

[0021] Based on the policy lookup table, look up the error handling policy corresponding to the determined common error code;

[0022] Judge whether the execution times of the error handling policy exceed a preset number threshold within a preset time period;

[0023] If so, reset the hardware device corresponding to the module that sends the exception information;

[0024] If not, execute the error handling policy.

[0025] In some embodiments, the control module communicates with multiple media data processing modules through PCIE links respectively; the method further includes the following steps:

[0026] Monitor the load status of each PCIE link, and determine the sampling frequency of the media data processing module corresponding to the PCIE link according to the load status;

[0027] Based on the sampling frequency of each media data processing module, obtain the sampling data from the media data processing module through the PCIE link.

[0028] In some embodiments, the multi-modal media data acquisition module includes multiple video acquisition devices; the media data processing module is configured to obtain the control instruction analysis result by the following steps:

[0029] Extract features from the video data of a single video acquisition device respectively;

[0030] Input the features of each video acquisition device into the gesture classification model respectively to obtain the gesture category corresponding to a single video acquisition device and the confidence level belonging to the gesture category;

[0031] Take the gesture category and confidence level corresponding to each video acquisition device as the control instruction analysis result and send it to the control module;

[0032] The control module sends the control instruction to the controlled device based on the control instruction analysis result determined by the media data processing module by the following steps:

[0033] Receive the analysis result of the control instruction, and use the confidence level as the weight of the corresponding video acquisition device;

[0034] Judge whether a gesture is detected according to the gesture category;

[0035] If the number of times a gesture is detected within a preset time period is greater than the preset gesture number threshold, adjust the weights of each video acquisition device to suppress the maximum value in the weights;

[0036] Perform weighted averaging on the gesture categories corresponding to multiple video acquisition devices to determine the average gesture category, generate a control instruction according to the average gesture category, and send it to the controlled device.

[0037] In some embodiments, the multimodal media data acquisition module includes multiple video acquisition devices; the media data processing module is configured to obtain the control instruction analysis result by the following steps:

[0038] Extract features from the video data of a single video acquisition device respectively;

[0039] Cluster the features of multiple video acquisition devices, select the group of features with the most clustering features and integrate them to obtain input features, input the input features into the gesture classification model, obtain the gesture category with the highest confidence level, and use the gesture category as the control instruction analysis result;

[0040] The control module receives the control instruction analysis result, generates a control instruction according to the gesture category, and sends it to the controlled device.

[0041] The second aspect of the present application also provides a device control system based on multimodal multi-core heterogeneous, including:

[0042] A control module for implementing the above-mentioned device control method based on multimodal multi-core heterogeneous;

[0043] A media data processing module, which has a different processing core from the control module and is used to analyze and obtain the control instruction analysis result based on the received data or data packet.

[0044] The third aspect of the present application also provides a device control device based on multimodal multi-core heterogeneous, including:

[0045] A processor;

[0046] A memory, in which executable instructions of the processor are stored;

[0047] Wherein, the processor is configured to execute the steps of the above-mentioned device control method based on multimodal multi-core heterogeneous by executing the executable instructions.

[0048] The fourth aspect of the present application further provides a computer program product, including a computer program, which when executed by a processor implements the steps of the above-mentioned device control method based on multi-modal multi-core heterogeneous architecture.

[0049] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit the present application.

[0050] The device control method, system, device and product based on multi-modal multi-core heterogeneous architecture of the present application have the following beneficial effects:

[0051] By adopting the device control method of the present application, the media data obtained by the multi-modal media data acquisition module is quickly and efficiently sent to the media data processing module, so that the media data processing module can process the media data in a timely manner and generate control instructions for controlling the controlled device. By collecting, transmitting and processing multi-modal data, the problem that is easily affected by the environment in the case of a single modality is solved, so that the control method can be applied to various scenarios and the accuracy of control instruction recognition is improved; when transmitting data from the control module to the media data processing module, a data communication method under a multi-core heterogeneous architecture is provided through the cooperation of the soft interrupt mechanism and the hard interrupt mechanism, which is not affected by the difference in the concurrent processing speed of different cores, and thus the problem of backlog of data transmission tasks is alleviated, and the delay problem caused by directly using the hard interrupt to read data from the data buffer is avoided, thereby improving the efficiency of data transmission, meeting the real-time requirements of high-concurrency data transmission when accessing multi-modal data, and further improving the efficiency from the time when the user gives an action or voice instruction to the response of the controlled device, and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] By reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings, other features, objects and advantages of the present application will become more apparent.

[0053] Figure 1 is a flowchart of the device control method based on multi-modal multi-core heterogeneous architecture according to an embodiment of the present application.

[0054] Figure 2 is a block diagram of the device control system based on multi-modal multi-core heterogeneous architecture according to an embodiment of the present application.

[0055] Figure 3 is a schematic diagram of the structure of the device control system based on multi-modal multi-core heterogeneous architecture according to an embodiment of the present application.

[0056] Figure 4 is a timing diagram of generating control instructions based on multi-modal media data and controlling a controlled device according to an embodiment of the present application.

[0057] Figure 5It is a timing diagram for data and command transmission in an embodiment of the present application.

[0058] Figure 6 It is an interaction timing diagram of a low-latency processing system in an embodiment of the present application.

[0059] Figure 7 It is a timing diagram for processing abnormal information of a device control system based on multi-modal multi-core heterogeneity in an embodiment of the present application.

[0060] Figure 8 It is a timing diagram for log processing in an embodiment of the present application.

[0061] Figure 9 It is a timing diagram for viewing the current state of an audio data processing module in an embodiment of the present application.

[0062] Figure 10 It is a timing diagram for verifying encryption and decryption results in an embodiment of the present application.

[0063] Figure 11 It is a timing diagram for the heartbeat mechanism of driver and firmware communication in an embodiment of the present application.

[0064] Figure 12 It is a schematic structural diagram of a device control device based on multi-modal multi-core heterogeneity in an embodiment of the present application. Detailed implementation manners

[0065] Now, example embodiments will be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0066] In addition, the accompanying drawings are only schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware units or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. Although the terms "first" or "second" etc. are used in this specification to represent certain features, they are only for the purpose of indication and do not limit the quantity and importance of the specific features.

[0067] The flowchart shown in the accompanying drawings is only an exemplary illustration and does not necessarily include all steps. For example, some steps can be further decomposed, while some steps can be combined or partially combined. Therefore, the actual execution order may be changed according to the actual situation.

[0068] To solve the technical problems in the prior art, the present application provides a device control method based on multi-modal multi-core heterogeneous architecture, which is used to quickly and efficiently send the media data obtained by the multi-modal media data acquisition module to the media data processing module, so that the media data processing module can process the media data in a timely manner and generate control instructions for controlling the controlled device. This control method is implemented based on a control module, and the control module and the media data processing module have different processing cores, constituting a multi-core heterogeneous device control system architecture.

[0069] As Figure 1 shown, an embodiment of the present application provides a device control method based on multi-modal multi-core heterogeneous architecture, and the method includes the following steps:

[0070] S100: Obtain the data to be transmitted from the multi-modal media data acquisition module and store it in the data buffer; this data buffer is the data buffer set in the control module;

[0071] Herein, the data to be transmitted is the multi-modal media data collected by the multi-modal media data acquisition module; multi-modal refers to different categories of media data, such as including visual modality (video data) and auditory modality (audio data). By combining video data and auditory data to analyze the analysis result of the control instruction, the problem that a single modality is easily affected by the external environment is avoided, and the accuracy of recognizing the user control instruction is improved; the multi-modal media data includes, for example, video data and audio data, and the multi-modal media data acquisition module includes, for example, a camera, a microphone, etc.;

[0072] S200: Every first preset time, the soft interrupt unit judges the backlog status of the data transmission tasks in the data buffer. If there is a backlog, all the data to be transmitted in the data buffer is packed and stored in the specified storage area. If there is no backlog, the data to be transmitted in the data buffer is sequentially read and stored in the specified storage area;

[0073] When the multimodal media data includes video data and audio data, the media data processing module includes, for example, an audio data processing module and a video data processing module. The control module generally uses a CPU (Central Processing Unit) core to execute the relevant processes in the control flow. The audio data processing module generally uses a DSP (Digital Signal Processor) core, and the video data processing module generally uses an NPU (Neural Processing Unit) core and a GPU (Graphics Processing Unit) core. Therefore, in the process of interaction between the control module and the media data processing module, multi-core heterogeneous communication is involved. In order to improve the real-time performance of data communication in the multi-core heterogeneous architecture, a soft interrupt unit is set in the control module to solve the problem of data transmission backlog and greatly improve the efficiency of data communication;

[0074] S300: Every second preset time, the hard interrupt unit reads the information to be transmitted from the specified storage area. The information to be transmitted includes the data or data packet to be transmitted and sends it to the media data processing module. The media data processing module and the control module have different processing cores, and the media data processing module is configured to analyze the received data or data packet to obtain the control instruction analysis result;

[0075] Both the soft interrupt unit and the hard interrupt unit are provided in the control module. The control strategies of the soft interrupt unit and the hard interrupt unit are both timing tasks. That is, the timing task of the soft interrupt unit is executed every first preset time. Its timing task is used to read the data in the data buffer into the specified storage area. The specified storage area is, for example, the cache area (high-speed cache area) of the soft interrupt unit. The timing task of the hard interrupt unit is executed every second preset time. Its timing task is used to read and send the data in the specified storage area to the media data processing module. The hard interrupt unit realizes the data transmission between the control module and the media data processing module. The time lengths of the first preset time and the second preset time can be set according to needs. For example, the first preset time is set to be greater than the second preset time, that is, the time granularity for the soft interrupt unit to read data is greater than that of the hard interrupt unit to read data. Or, the time lengths of the first preset time and the second preset time are the same. Or, the first preset time can also be set to be less than the second preset time. In this embodiment, the second preset time of the hard interrupt unit is determined according to the main frequency of the RTC (Real_Time Clock) chip clock. Taking the main frequency of the RTC chip clock as 37M as an example, the second preset time of the hard interrupt unit is, for example, 10 times the chip clock interval time, that is, (1 / 37M)*10s. Then the second preset time is about 300μs. The first preset time of the soft interrupt unit is determined according to the main frequency of the CPU chip clock. The first preset time is about 1ms. There may be about 5% jitter during operation. During system operation, the soft interrupt unit and the hard interrupt unit perform corresponding timing tasks simultaneously without external event triggering;

[0076] S400: Send the control instruction to the controlled device based on the analysis result of the control instruction determined by the media data processing module.

[0077] By adopting the device control method of the present application, it is realized that the media data acquired by the multi-modal media data acquisition module is quickly and efficiently sent to the media data processing module, so that the media data processing module can process the media data in a timely manner and generate control instructions for controlling the controlled device. By collecting, transmitting, and processing multi-modal data, the problem of being easily affected by the environment in the case of a single modality is solved, so that the device control method can be applied to various scenarios and the accuracy of control instruction recognition is improved; when transmitting data from the control module to the media data processing module, a data communication method under a multi-core heterogeneous architecture is provided through the cooperation of the soft interrupt mechanism and the hard interrupt mechanism, which is not affected by the difference in concurrent processing speeds of different cores, realizes and alleviates the problem of backlog of data transmission tasks, and avoids the delay problem caused by directly using the hard interrupt to read data from the data buffer, thereby improving the efficiency of data transmission, meeting the real-time requirements of high-concurrency data transmission when accessing multi-modal data, and further improving the efficiency from when the user gives an action or voice instruction to the response of the controlled device, and enhancing the user experience.

[0078] In this embodiment, the device control method further needs to transmit commands to the media data transmission module. For example, audio processing parameters are sent to the audio data processing module, video processing parameters are sent to the video data processing module, etc. The information to be transmitted sent by the hard interrupt unit to the media data processing module also includes commands to be transmitted. The device control method further includes the control module sending commands to be transmitted to the media data processing module by the following steps:

[0079] Obtain the commands to be transmitted and store them in the command buffer. Among them, if there is a backlog of command transmission tasks in the command buffer and the current number of command transmission tasks does not exceed the maximum command number threshold, the capacity of the command buffer is expanded. If the number of command transmission tasks in the command buffer exceeds the maximum command number threshold, the capacity is not expanded and an exception notification is issued;

[0080] Every first preset time, the soft interrupt unit reads the commands to be transmitted from the command buffer and stores them in the specified storage area;

[0081] When, through step S300, the hard interrupt unit reads the information to be transmitted from the specified storage area, it reads the data to be transmitted, data packets, and / or information in the specified storage area.

[0082] Therefore, the present application further realizes command transmission in a multi-core heterogeneous architecture through the secondary interrupt mechanism of the soft interrupt unit and the hard interrupt unit, avoids the influence of the difference in processing speeds between different hardwares in a multi-core heterogeneous scenario, improves the real-time performance of command transmission, makes the command transmission between the control module and the media data processing module smoother and more efficient, and controls the processing parameters of the media data processing module more timely, which is conducive to improving the accuracy of the subsequent media data processing module in analyzing media data.

[0083] In this embodiment, in step S300, the hard interrupt unit reads the information to be transmitted from the specified storage area, including the following steps:

[0084] The hard interrupt unit determines the priority of the information to be transmitted in the specified storage area;

[0085] The hard interrupt unit sequentially reads the information to be transmitted in the specified storage area according to the priority order and sends it to the media data processing module; among them, the hard interrupt unit reads the information to be transmitted with a priority higher than the preset level threshold and sequentially sends it to the media data processing module, and packs the information to be transmitted with a priority lower than or equal to the preset level threshold and sends it to the media data processing module;

[0086] Therefore, the present application further adds the judgment and discrimination processing of the information reading priority in the hard interrupt unit, preferentially reads the information with high priority, so that the information with high priority, that is, the information with higher real-time requirements, can be transmitted to the media data processing module faster. For the information with lower priority, in order to avoid backlog, it can be packed and sent to the media data processing module, which can also ensure the transmission efficiency of the information with lower priority. For example, the information to be transmitted is divided into three priorities: high priority, medium priority, and low priority. The command to be transmitted is set as high priority, and the data or data packet to be transmitted is set as medium priority or low priority. The medium priority and low priority are judged according to the number of data transmission tasks in the current specified storage area. When the number of data transmission tasks in the current specified storage area is greater than the first quantity threshold, it is considered that the data or data packet to be transmitted in the current specified storage area is low priority, otherwise it is considered that the data or data packet to be transmitted in the current specified storage area is high priority. The hard interrupt unit sequentially reads the information to be transmitted in the specified storage area according to the priority order and sends it to the media data processing module, including: the hard interrupt unit first reads the command to be transmitted with high priority and sends it to the media data processing module one by one, then the hard interrupt unit sends the data or data packet to be transmitted with medium priority to the media data processing module one by one, and then the hard interrupt unit packs the data or data packet to be transmitted with low priority and sends it to the media data processing module.

[0087] As Figure 2 shown, the embodiment of the present application also provides a device control system based on multi-modal multi-core heterogeneous, including: a control module M100, which is used to implement the above-mentioned device control method based on multi-modal multi-core heterogeneous; a media data processing module M200, which has a different processing core from the control module M100 and is used to analyze the received data or data packet to obtain the analysis result of the control instruction. When the device control system runs, it can implement the steps of the above-mentioned device control method and achieve the technical effects of the above-mentioned device control method, which will not be elaborated here.

[0088] The device control method and device control system of the present application can be applied to various control scenarios. For example, it can be applied to the scenario where a teacher controls a computer device with actions or voice in the classroom. That is, the controlled device is a computer device for playing courseware. Multimodal media data can be collected through cameras and microphones installed at multiple positions in the classroom for analyzing the teacher's instructions. Or it can be applied to the scenario where an operator controls production equipment with actions or voice in the factory. That is, the controlled device is production equipment. Multimodal media data can be collected through cameras and microphones installed at multiple positions in the workshop for analyzing the operator's instructions.

[0089] The following takes the application of the device control method and device control system to the scenario where a teacher controls a computer device with actions or voice in the classroom as an example for illustration. Figure 3It is a schematic structural diagram of a device control system based on multi-modal multi-core heterogeneous in an embodiment of the present application. Among them, MSC (Main System Control) represents the control module, which is a hardware module responsible for multi-modal media data (access of multiple channels of audio and video data), decoding, and task scheduling, and is deployed on the AI (Artificial Intelligence) fusion host (the hardware main body for fusion control of multi-modal fusion). MDC (Main Data Control) represents the multi-channel data module, which is a software module in the control module, responsible for specific data processing and balanced control according to the resource consumption of multi-channel processing, and directly outputs the processed data to external devices through a hardware interface. ADA (Audio Data Aggregation) represents the audio data aggregation unit, which is a software module in the control module, used to collect and aggregate audio data, and buffer audio data according to the data consumption situation of MDC. IPC (IP Camera) represents an IP camera, used to collect video data and audio data. RTC (Real-Time Communication) represents the interactive audio collection module, used to collect interactive audio data. TS (Tip-Sleeve, a type of audio interface) represents the audio microphone interface, used to collect audio data, and the microphone includes types such as wireless microphones, desktop microphones, and network digital microphones. DEC (Decoder) represents the data decoding module, used to decode digital audio data and video data into YUV images or audio data. ENC (Encoder) represents the data encoding module, used to encode analog audio data or uncompressed YUV images (YUV is a color encoding method) into compressed data. HDMI_IN represents receiving audio data and video data through the HDMI (High Definition Multimedia Interface) hardware interface, and IN represents input. CMS (Center Management System) corresponds to the task management module, which is a software module in the control module, used for message reception and processing. In Figure 3Among them, MUX (Multiplexer) represents a multiplexer, Demux (Demultiplexer) represents a demultiplexer, MIC represents a microphone, AUD represents audio data, and converting an image to the Mat format means converting common image files (such as JPEG, PNG, etc.) into a.mat format file dedicated to MATLAB. The interfaces at Head-up TV 1 and Head-up TV 2 can be correspondingly set to the interfaces of computer PC1 or PC2. The interfaces at the live / on-demand recording device correspondingly include the interfaces of computer PC1, PC2, wireless microphone, desktop microphone, network digital microphone, etc. The interfaces at personal computer PC correspondingly include the interfaces of PC1, PC2, wireless microphone, desktop microphone, network digital microphone, etc. The interfaces at RTC_OUT_1 correspondingly include the interfaces of PC1, PC2, wireless microphone, desktop microphone, etc.

[0090] In Figure 3 the embodiment of, the control module obtains 8 streams of code (such as a video code stream with a resolution of 1080P), 3 streams of interactive RTC audio data, 3 streams of audio microphone and HDMI audio data from the multi-modal media data acquisition module (IPC, microphone, RTC, HDMI device, etc.). Through network audio data aggregation and analog audio data aggregation to MDC, after MDC scheduling, it is sent to the audio data processing module (Audio Data Processing Unit, ADPU) and the video data processing module (Video Data Processing Unit, VDPU) for multi-modal media data processing, and based on the control instructions obtained from the analysis of ADPU and VDPU, it reversely controls the computer device (the controlled device, that is Figure 3 the PPT teaching computer in) through HDMI data. This device control system upgrades the single-modal acquisition control method to a multi-modal multi-channel (audio and video) data source control method. The control module communicates with the media data processing module (ADPU and VDPU) through a PCIE (Peripheral Component Interconnect Express, high-speed serial computer expansion bus) link, and data transmission with the media data processing module is carried out through the low-latency and high-concurrency of the PCIE high-speed bus of the board multi-processor, and the control instructions are transmitted to the controlled device through the HDMI data cable, so as to ensure the timeliness and accuracy of the control instructions.

[0091] In one implementation, when the device control method and device control system are applied to the scenario where a teacher controls a computer device with gestures or voice in the classroom, high-definition cameras are deployed at multiple locations in the classroom (such as the front, rear, and top of the classroom), and multiple microphone arrays are arranged. A PPT teaching computer is set at the podium. A ceiling-mounted TV can also be set in the classroom to synchronously play the display screen in the PPT teaching computer. The teacher's lecture process can also be live-streamed, and students can synchronously view the display screen in the PPT teaching computer through live-streaming devices, PCs (Personal Computers), RTC devices, etc. The teacher does not need to always stand in front of the PPT teaching computer during the lecture, but can walk around freely in the classroom. The teacher's gestures or voice are captured by the cameras and microphone arrays, and then the PPT teaching computer is controlled. The specific implementation process is as follows:

[0092] When the teacher makes a control gesture in the classroom, multiple cameras collect high-definition video data. The control module obtains the video data from the multiple cameras, and then sends the video data to the video data processing module. The NPU computing unit of the video data processing module performs gesture classification (such as waving, pointing, etc.), and then the control module generates a control instruction, and then outputs the control instruction to the PPT teaching computer. When the teacher speaks a voice command in the classroom, multiple microphone arrays collect audio data. The control module obtains the PCM (Pulse Code Modulation) audio data from the multiple microphone arrays, and then sends the audio data to the audio data processing module. The DSP chip of the audio data processing module performs noise reduction and spatial different voiceprint matching on the collected audio data, and can combine with the ARM CPU to implement voice decoding and control instruction parsing in the NPU computing unit (such as "PPT page down", "play video", etc.), determine the control instruction analysis result. The control module generates a control instruction according to the control instruction analysis result, and then outputs the control instruction to the PPT teaching computer. The PPT teaching computer executes the actions corresponding to the control instructions, such as turning the PPT page, playing a video, etc. The control module obtains the playback data in the PPT teaching computer (including the display screen data in the PPT teaching computer, and when playing a video in the PPT teaching computer, it also includes the playback audio in the PPT teaching computer), and then sends the playback data in the PPT teaching computer to the ceiling-mounted TV, PC device, on-demand device, RTC device, etc. Students can synchronously view the playback content in the PPT teaching computer through other channels.

[0093] Figure 4 It is a timing diagram for generating control instructions based on multi-modal media data and controlling a controlled device according to an embodiment of the present application. The following is combined with Figure 4This article will introduce the implementation method of the control process. Among them, the process of generating control instructions based on multi-modal media data and controlling the controlled device is divided into two main parts: The first part is the access of media data, that is, transmitting the media data of the multi-modal data acquisition module to the control module and reporting the status, ensuring that the CMS task management module can timely obtain and process the data of multi-channel cameras and multi-channel microphones; The second part is the analysis and control of multi-channel audio and video data, that is, sending the multi-modal data to the ADPU and VDPU for calculation and processing, and controlling the PPT teaching device, which is used for the analysis and control of multi-channel audio and video data, including audio processing, gesture recognition, and the control of the PPT teaching computer, to achieve more complex functions and interactions.

[0094] As Figure 4 shown, the first part of the control process includes the following steps:

[0095] Step 1: Analyze channel addition, deletion, or modification: The CMS task management module detects a change in the configuration of the analysis channel, which may include adding, deleting, or modifying channels.

[0096] Step 2: Modify the analysis channel: The CMS task management module updates the configuration of the analysis channel according to the detected modification.

[0097] Step 3: Push the original code stream to the MSC control module: The multi-channel camera pushes the original video code stream (video data) to the MSC control module.

[0098] Step 4: Report the connection status: The multi-channel camera reports its connection status to the CMS task management module.

[0099] Step 5: Push the multi-channel microphone audio data to the MSC control module.

[0100] Step 6: Push the audio status to CMS: The MSC pushes the audio status information to the CMS task management module.

[0101] As Figure 4 shown, the second part of the control process includes the following steps:

[0102] Step 1: Analyze channel modification: The CMS task management module detects a change in the configuration of the analysis channel.

[0103] Step 2: Push multi-channel audio Raw data (original data): The multi-channel microphone pushes the original audio data to the MSC control module.

[0104] Step 3: Convert different sound sources into digital array information and perform AFC (Adaptive Feedback Cancellation) echo suppression: The ADPU / VDPU converts the audio data of different sound sources into digital array information and performs AFC echo suppression processing to obtain the audio transcription result, which is used as the analysis result of the control instruction. The audio transcription result includes the text transcribed from the audio data.

[0105] Step 4: Push the audio transcription result to the CMS: The ADPU / VDPU pushes the processed audio transcription result to the CMS task management module.

[0106] Step 5: Compare with the behavior vector table and generate control instructions: The CMS task management module compares the audio transcription result with the behavior vector table to determine the type of control instruction corresponding to the audio data, and generates control instructions according to the type of control instruction. For example, if the audio transcription result is "page turning", the control instruction is to turn the current playing PPT by one page; if the audio transcription result is "stop playing", the control instruction is to stop the current playing video.

[0107] Step 6: Push YUV data to the VDPU: The CMS task management module pushes the video data in YUV format to the VDPU.

[0108] Step 7: Identify the gesture category according to the video data: The VDPU performs feature acquisition and detection on different video images, identifies the gesture category of the human body, and takes the gesture category and the extracted features as the gesture analysis result, and takes the gesture analysis result as the analysis result of the control instruction.

[0109] Step 8: Push the gesture analysis result to the CMS: The VDPU pushes the gesture analysis result to the CMS task management module.

[0110] Step 9: Generate control instructions: After receiving the gesture analysis result, the CMS task management module generates corresponding control instructions.

[0111] Step 10: Control the PPT teaching computer through the HDMI high-speed signal line: The CMS task management module sends a control signal to the PPT teaching computer through the HDMI high-speed signal line to realize the control of the PPT playing process.

[0112] In this embodiment, in step 8, the video data processing module VDPU is configured to adopt a two-level gesture evaluation strategy to determine the gesture category. Specifically, the media data processing module obtains the analysis result of the control instruction through the following steps of the first-level gesture evaluation strategy and the second-level gesture evaluation strategy.

[0113] The first-level gesture evaluation strategy (weighted average after classification by a single video acquisition device) includes the following steps:

[0114] The VDPU extracts features from the video data of each individual video acquisition device respectively; for example, the video data of an individual video acquisition device is input into a hand key point detection model, and the hand key point feature data is detected and used as the feature corresponding to the individual video acquisition device; the video acquisition device is, for example, an IPC, a camera, a network video transmission device, etc.

[0115] The features of each video acquisition device are respectively input into a gesture classification model to obtain the gesture category corresponding to the individual video acquisition device and the confidence level belonging to the gesture category. The hand key point detection model and the gesture classification model can adopt existing machine learning models.

[0116] The gesture category and confidence level corresponding to each video acquisition device are used as the control instruction analysis result and sent to the CMS task management module of the control module.

[0117] Correspondingly, in step 10, the CMS task management module of the control module uses the following steps to send the control instruction to the controlled device (PPT teaching computer) based on the control instruction analysis result determined by the VDPU:

[0118] Upon receiving the control instruction analysis result, the confidence level is used as the weight of the corresponding video acquisition device.

[0119] Upon receiving the control instruction analysis result, the confidence level is used as the weight of the corresponding video acquisition device.

[0120] It is judged whether a gesture is detected according to the gesture category; for example, if the gesture category corresponding to at least one video acquisition device is a certain specific gesture detected, it is considered that a gesture is detected.

[0121] If the number of times a gesture is detected within a preset time period is less than or equal to the preset gesture number threshold, the gesture categories corresponding to multiple video acquisition devices are directly weighted and averaged to determine the average gesture category, and the average gesture category and the features of the video acquisition device are used as the control instruction analysis result.

[0122] If the number of detected gestures within a preset time period is greater than the preset gesture number threshold, it indicates that the user has made multiple actions within a relatively short time interval, and there may be cases of previous misjudgment or missed judgment. For example, if the user finds that the computer device does not respond after a wave of the hand, the user will quickly wave the hand again. The preset time period can be set to a relatively small value. For example, if multiple gestures made by the user are detected within 30s, it may be that the user's initial gesture did not receive a correct response. Then, it is necessary to adjust the weights of each video capture device to reduce the possibility of misjudgment. When adjusting the weights, suppress the maximum value in the weights so that the maximum value in the weights of each video capture device approaches the median value, avoiding the influence of the maximum weight value on the weights.

[0123] In one implementation, when adjusting the weights, multiply the weights of each video capture device by an adjustment coefficient to obtain the adjusted weights. The adjustment coefficient is obtained by querying a coefficient table, which is a corresponding table of the weights before adjustment and the adjustment coefficients. In this coefficient table, for the weights before adjustment greater than 0.5, the corresponding adjustment coefficient is less than 1, so that for the weights before adjustment greater than 0.5, the adjusted weights approach 0.5. For the weights before adjustment less than 0.5, the corresponding adjustment coefficient is greater than 1, so that for the weights before adjustment less than 0.5, the adjusted weights approach 0.5. When the weight before adjustment is 0.5, the adjustment coefficient is 1. Further, for the weights before adjustment greater than 0.5, the larger the weight before adjustment, the smaller the corresponding adjustment coefficient to achieve maximum value suppression. For the weights before adjustment less than 0.5, the smaller the weight before adjustment, the larger the corresponding adjustment coefficient to achieve minimum value suppression;

[0124] After obtaining the adjusted weights, perform weighted averaging on the gesture categories corresponding to multiple video capture devices to determine the average gesture category, generate a control instruction according to the average gesture category, and send it to the controlled device.

[0125] The first-level gesture evaluation strategy of this application fully considers the differences in data collection between individual video capture devices, and comprehensively weights and averages the classification results of multiple video capture devices to obtain a more accurate gesture classification result. At the same time, when misjudgment or missed judgment may occur, the weight values of each video capture device are adjusted in a timely manner to reduce the misjudgment rate and missed judgment rate, and further improve the accuracy of gesture classification.

[0126] The second-level gesture evaluation strategy (realize gesture classification through feature-level fusion of multiple video capture devices) includes the following steps:

[0127] Extract features from the video data of individual video capture devices respectively; for example, input the video data of an individual video capture device into a hand key point detection model, and detect and obtain the hand key point feature data as the feature corresponding to the individual video capture device;

[0128] Cluster the features of multiple video capture devices, select the group of features with the most clustered features and integrate them to obtain input features. Input the input features into the gesture classification model to obtain the gesture category with the highest confidence, and use the gesture category as the analysis result of the control instruction.

[0129] For example, the input features are obtained by averaging each dimension in the group of features with the most clustered features to obtain the integrated input features. After inputting the input features into the gesture classification model and obtaining the confidence levels predicted by the gesture classification model for various gesture categories, the gesture category with the highest confidence is used as the analyzed gesture category.

[0130] Adopt the second-level gesture evaluation strategy. Correspondingly, in step 10, the CMS task management module of the control module receives the analysis result of the control instruction, generates a control instruction according to the gesture category, and sends it to the controlled device (PPT teaching computer).

[0131] The second-level gesture evaluation strategy of the present application performs feature-level fusion on the video data of multiple video capture devices and then classifies gestures, so that the input features of the gesture classification model can more accurately represent the hand key point features of the gestures made by the teacher, thereby obtaining more accurate gesture classification results.

[0132] The first-level gesture evaluation strategy and the second-level gesture evaluation strategy can be used in combination according to different gesture evaluation requirements. For example, when the algorithm accuracy requirement in the gesture evaluation strategy configuration of the device control system is low and the response speed is fast, the first-level gesture evaluation strategy is adopted. When the algorithm accuracy requirement in the gesture evaluation strategy configuration of the device control system is high and the response speed is not required, the first-level gesture evaluation strategy is used to extract the video data features of a single video capture device and then the second-level gesture evaluation strategy is adopted. The gesture evaluation strategy configuration can be configured by the user at the control module or automatically set according to the current misjudgment rate. For example, initially adopt the first-level gesture evaluation strategy. If after adjusting the weights, the number of times of detecting gestures within a preset time period is greater than the preset gesture number threshold, indicating that the misjudgment rate is still relatively high, then jump to the second-level gesture evaluation strategy and send a command to the VDPU to switch the gesture evaluation strategy.

[0133] Figure 5 It is a timing diagram for data and command transmission in an embodiment of the present application. The present application provides a data transmission method based on a multi-modal multi-core heterogeneous architecture. Corresponding to the above step S100, after data acquisition, it is written into the data buffer. Corresponding to the above steps S200 and S300, a software interrupt unit and a hardware interrupt unit are used to achieve asynchronous data transmission between the control module and the media data processing module. Specifically, a data writing unit, a data buffer, a command buffer, a software interrupt unit, and a hardware interrupt unit are set in the MDC.

[0134] In this application, the MDC dynamically allocates computing resources based on task priorities (for example, the audio acceleration image processing is performed by [specific component], and the CPU is responsible for audio analysis), reducing the communication latency between multiple cores. The PCIE bus is used as the data communication carrier for communication.

[0135] The content of the inter-core data communication in the asynchronous framework is as follows:

[0136] Data processed by the ADPU module: The ADPU needs to process audio data of 24 channels, 10 channels are used for audio data access, 10 channels are used for audio data output, 3 channels are used for control command communication, and 1 channel is used for heartbeat keep-alive connection.

[0137] After the ADPU starts working, it queries the buffer situation of 24 channels. If the data in the read buffer of one or more channels is ready and the write buffer of this channel is also allocated, the ADPU starts to process the data of these channels in sequence.

[0138] The ADPU moves the data in the read buffer into the internal data memory, and then moves it out to the write buffer after audio intelligent processing.

[0139] When all the data in the read buffer is read in, the ADPU will report an interrupt to the outside to notify the MDC that the read buffer can be released; when all the processing results of the ADPU are written into the write buffer, it will also report an interrupt to the outside to notify the MDC that the data has been processed, and the audio transcription result has been stored in the specified location, and the MDC can read the audio transcription result for subsequent processing.

[0140] The conventional method of multi-core notification for PCIE is that when a PCIE device generates an event, it notifies the CPU core through a hardware interrupt (such as MSI / MSI-X. MSI is Message Signaled Interrupt, which means that the device triggers a CPU interrupt by writing a specific message to a specific address. MSI-X is an extension and enhancement of MSI, with the full name Message Signaled Interrupts - Extended). Due to the limited number of hardware interrupt numbers, a single interrupt may need to correspond to multiple events. The control message and the bitstream message share a hardware interrupt, which affects the real-time performance of the data and the waiting problem of high-priority tasks. To solve this problem, this application uses a software interrupt unit and a hardware interrupt unit to achieve asynchronous data transmission between the control module and the media data processing module, improving the real-time performance of data transmission.

[0141] By adopting the above steps S200 and S300 and the steps of command transmission through secondary interrupts (soft interrupts and hard interrupts), every first preset time, the soft interrupt unit triggers batch processing, that is, triggers a custom soft interrupt Pcie_tasklet (reusing TASKLET_SOFTIRQ). Tasklet represents the soft interrupt delay mechanism in the interrupt processing mechanism, and SOFTIRQ represents soft interrupt (Soft Interrupt), delaying the event processing flow to be executed in the soft interrupt context, thereby reducing the time occupied by hardware interrupts. Through non-vector interrupt processing optimization, by adopting a non-vector interrupt architecture, the soft interrupt unit receives all transmission tasks through a single entry address, and then distinguishes specific event sources through software flag bits or event descriptors, reducing hardware dependence. In the implementation, two kernel buffer queues Queue are added, including a data buffer queue DataQueue and a command buffer queue CmdQueue). The task processing function of the soft interrupt unit reads data transmission tasks or command transmission tasks from the kernel buffer queue.

[0142] The following combines Figure 5 Specifically introduce the implementation processes of the soft interrupt unit and the hard interrupt unit.

[0143] I. Data transmission task processing flow:

[0144] Step 1: MDC writes data to the data buffer, corresponding to step S100. The data writing unit writes the data to be transmitted obtained from the multi-modal media data processing module into the data buffer, and creates a data transmission task in the data buffer;

[0145] When writing to the data buffer, a new optimization logic judgment is added: the data writing unit judges whether the number of data transmission tasks in the data buffer is greater than the maximum task number threshold. If so, sort the data transmission tasks by time, delete the data transmission task with the earliest addition time, and set the task backlog flag bit to True (true), otherwise set the data backlog flag bit to False (false);

[0146] Step 2: The soft interrupt Pcie_tasklet reads data from the data buffer, corresponding to step S200. The soft interrupt unit reads the data to be transmitted from the data buffer, and adds an optimization logic judgment at the soft interrupt unit. The soft interrupt unit, according to the task backlog identification status of the data buffer, if it is True, reads all the data to be transmitted in all the data buffers at one time, clears the data buffer queue, deletes the corresponding task node in the data buffer after reading the task, and stores the read data to be transmitted in a specified storage area after packing it into a data packet to be transmitted. If it is False, reads the data in the data buffer queue in the data buffer in sequence, stores it in the specified storage area, and deletes the corresponding task node in the data buffer.

[0147] II. Command Task Processing Flow:

[0148] Step 3: The MDC writes the command to the command buffer, i.e., the data writing unit obtains the command to be transmitted and stores it in the command buffer;

[0149] Step 4: When writing to the command buffer, add an optimization logic judgment: the data writing unit judges whether there is a backlog of command transmission tasks in the command buffer (the command transmission task is greater than or equal to the preset backlog quantity threshold) and the current command transmission task quantity does not exceed the maximum command quantity threshold (the maximum command quantity threshold is greater than the preset backlog quantity threshold). If so, set the task backlog flag bit of the data buffer to True, and expand the capacity of the command buffer to achieve dynamic expansion of the command transmission task, and notify the MDC of the backlog exception event in the command buffer. If the current command transmission task quantity in the command buffer exceeds the maximum command quantity threshold, do not expand the capacity, and notify the MDC of the backlog exception event in the command buffer; if the current command transmission task quantity is less than the preset backlog quantity threshold, do not expand the capacity and set the task backlog flag bit of the data buffer to False;

[0150] Step 5: The soft interrupt Pcie_tasklet reads the command from the command buffer: Every first preset time, the soft interrupt unit judges the status of the task backlog flag in the command buffer, reads the commands to be transmitted in sequence according to the command buffer sequence from the command buffer and stores them in the specified storage area, and deletes the corresponding task node in the command buffer.

[0151] III. The hard interrupt unit reads data and commands:

[0152] Step 6: The hard interrupt reads data, i.e., corresponding to step S300, the hard interrupt unit reads data, data packets or commands from the specified storage area.

[0153] In step S300, different from the conventional First - In - First – Out (FIFO) method, the hard interrupt unit uses a three - level priority logic judgment method to read data from the specified storage area and send it to the media data processing modules ADPU and VDPU. The three priorities are divided as follows:

[0154] High priority: First, read the commands to be transmitted in the specified storage area one by one and send them to the media data processing module.

[0155] Medium priority: When there is no backlog of data or data packets to be transmitted in the specified storage area, read the data to be transmitted in the specified storage area one by one and send it to the media data processing module.

[0156] Low priority: When there is a backlog of data or data packets to be transmitted in the specified storage area, all the data or data packets to be transmitted in the specified storage area are retrieved, packed, and sent to the media data processing module.

[0157] Therefore, the present application separates the configuration control command transmission and data transmission processes in the multi-core heterogeneous communication process, and dynamically adjusts the priority of information transmission by judging the information category through the hard interrupt unit, thereby ensuring high-bandwidth transmission. Under the high processing ability of the ADPU, the overall performance will not be affected by the throughput bandwidth limitation of the PCIE bus.

[0158] Figure 6 It is the interaction timing diagram of the low-latency processing system according to an embodiment of the present application. General structured data and bit stream data are bit streams without data headers in the SoC (System on Chip) system. When the ADPU processes this kind of bit stream, all the data in the entire read buffer is processed, and the result is stored in the write buffer. The process is relatively simple. The services (ADA and MDC) in the control module need to configure the following information for the ADPU: 1. Working mode: audio quality processing (enhanced), audio quality processing and voice content recognition; 2. Audio quality algorithm selection, AEC / AGC, and AudioReg, where AEC represents Acoustic Echo Cancellation, AGC represents Automatic Gain Control, and AudioReg represents Audio Recognition; 3. Packet mode selection: In the full-modal working mode, the initial header information needs to be passed in to record the type of transmitted data (such as PCM audio format and sampling rate, etc.). In the recognition working mode, no file information needs to be recorded, and only the fixed single-channel audio data of 16 kHz needs to be transmitted. These configuration information are transmitted to the ADPU in the form of commands.

[0159] To prevent the appearance of pseudo packet headers in the data processing results, the ADPU needs to turn on or off the bitstuffing module function according to the situation. Among them, the bitstuffing module is implemented by hardware and added to the data path between the bus and the ADPU. Bitstuffing is divided into the operation of writing the ADPU result to bitstuffing and the operation of removing bitstuffing when the MSC control module reads the processing result from the DDR. The work of the bitstuffing module is managed through the data cache of the ADPU, and the bitstuffing module is switched on and off as needed. First, configure the starting address of bitstuffing, configure the length of the data before passing through the bitstuffing module, and query the length of the data after bitstuffing is completed. When an exception error occurs in the bitstuffing module, the bitstuffing module needs to be reset, the prefetched data in the bitstuffing module needs to be cleared, and reconfigured. By adopting this method, it is beneficial to the hyper-threading processing inside the ADPU, reduces the complexity of commands by splitting commands, and improves the command processing speed; considering the premise of giving priority to ensuring the speed of processing audio data, the consistency and compliance of data processing are verified through dynamic data judgment and the global bitstuffing status.

[0160] The device control system of this application integrates multiple processing cores and realizes data and command communication between different modules through the PCIE data bus. With different business scenarios, the device control system of this application also involves scenarios such as exception information processing, log processing, viewing the current status of the audio data processing module, encryption and decryption result verification, and the heartbeat mechanism scenario of driver and firmware communication.

[0161] Figure 7 It is the timing diagram of the exception information processing of the device control system based on multi-modal multi-core heterogeneous in an embodiment of this application. Among them, SDK represents the software development kit, APP represents the application program, Fireware represents the firmware, ioctl represents the input / output control, and API represents the application programming interface. In this embodiment, a common error code lookup table for the control module and the media data processing module is pre-constructed, and a policy lookup table corresponding to the common error code and the error handling policy is pre-constructed. The device control method further includes the following steps:

[0162] The MDC receives exception information from a control module or a media data processing module; the exception information includes, for example, abnormal parameter settings caused by incorrect formats found during ADPU data processing, which affect audio data processing, stack processing exceptions encountered during hardware operation, and problems encountered during the processing of the driver HAL (Hardware Abstraction Layer) during ADPU system integration, such as resource contention; the exception information received by the MDC can be the exception information sent in real time when the ADPU / VDPU fails, or the exception information obtained from the non-real-time data analysis of the ADPU / VDPU. The non-real-time data includes, for example, the real-time resource status (memory and processor usage rate) of the ADPU / VDPU and the data processing status (receiving / processing / feedback), etc.;

[0163] The MDC determines the general error code corresponding to the exception information based on a general error code lookup table;

[0164] This general error code lookup table is a standardized error code lookup table comprehensively constructed considering the characteristics of different modules and different hardware devices applied by different modules in a heterogeneous multi-core system, establishing a unified error classification system for the heterogeneous multi-core system. The error categories may involve hardware errors, communication timeouts, data anomalies, etc. Different errors in the heterogeneous multi-core system are uniformly classified and encoded, and during lookup, fast cross-core retrieval of error codes is achieved through a hash table;

[0165] The MDC determines the error handling strategy corresponding to the general error code based on a policy lookup table;

[0166] After determining the error code, the MDC automatically routes to the policy lookup table to find the error handling strategy, that is, the strategy on how to solve the exception corresponding to the error code through software control methods; the policy lookup table stores the mapping relationship between the error code combination and the error handling strategy. Each error code combination (a combination of one or more error codes) corresponds to an exception event, and this exception event corresponds to one or more error handling strategies; there can be multiple error handling strategies, such as single-channel buffer overwrite discard, multi-channel buffer overwrite discard, discard time period, single-channel buffer emptying, multi-channel buffer emptying, algorithm core reset, time calibration synchronization, etc.; the mapping relationship between the error code combination and the error handling strategy in the policy lookup table can be initially configured by the user and dynamically updated later;

[0167] The MDC determines whether the execution count of the error handling strategy exceeds a preset count threshold within a preset time period;

[0168] If so, it means that the execution of the error handling strategy of the software multiple times has not worked, so the hardware device corresponding to the module that sent the exception information is reset;

[0169] If the answer is no, then an error handling strategy is executed. After the MDC locates the error handling strategy, it sends the error handling strategy to the module where the exception occurred. For example, for audio processing errors, the MDC sends the error handling strategy to the ADPU, and for video processing errors, the MDC sends the error handling strategy to the VDPU.

[0170] Through this method for processing exception information, unified processing of exception information in a heterogeneous multi-core system is achieved, and fast cross-core parsing of error codes is realized in combination with a hash table. After the error code is located, it is quickly routed to the policy lookup table to quickly obtain the error handling strategy corresponding to the exception information, improving the efficiency of exception information processing. By counting the execution times of the error handling strategy within a certain period of time, the historical success rate is statistically calculated, and when software processing fails multiple times, hardware reset is triggered to increase the probability of successful exception handling.

[0171] The policy lookup table in this application is a dynamically changing lookup table. After initially deploying the device control system, the MDC and the media data processing modules (ADPU and VDPU) negotiate on the error handling strategies therein to determine the possible error handling strategies that the media data processing modules may execute. During the operation of the device control system, the user can adjust the error handling strategies at the control module end according to the operation status of the device control system, such as adding error handling strategies, deleting error handling strategies, modifying the mapping relationship between error handling strategies and error code combinations, etc., thereby updating the policy lookup table. After the policy lookup table is updated, the control module notifies the media data processing modules (ADPU and VDPU) of the updated error handling strategies, and the error handling strategies are also updated at the media data processing module end. The control module can be configured to notify the media data processing modules to update according to the latest error handling strategies each time the hardware device carrying the control module starts or the firmware is upgraded, so that the policy lookup table can dynamically adapt to the current operation status of the device control system.

[0172] Figure 8It is a timing diagram of log processing in an embodiment of the present application. Among them, cfg represents configuration, and info / warn / dbg / err respectively represent functions of different levels (information / warning / debug / error). By setting the identification bits of the hardware (such as the switching state of registers, the state switching of data interaction of the AMMU (Advanced Memory Management Unit)), and the states during the operation of the driver HAL in the ADPU system integration, probe points can be reserved. Different probe point positions and types can be dynamically adjusted according to the device operation states (Error / Debug / Warn / Info / All, corresponding to error / debug / warning / information / all). In this embodiment, the device control method further provides steps for dynamically adjusting the sampling frequency of the probe points. Specifically, it includes: monitoring the load status of each PCIE link, determining the sampling frequency of the probe points of the media data processing module corresponding to the PCIE link according to the load status; based on the sampling frequencies of the probe points of each media data processing module, obtaining the probe point data from the media data processing module through the PCIE link. For example, when the communication link load of the ADPU is high, sparse sampling is used for the probe points of the APDU, and when the communication link load of the ADPU is low, full-scale sampling is used for the probe points of the ADPU. Since PCIE is a channel for multi-core hardware communication, it has the characteristics of many channels, many categories (large buffer data blocks of video images and small audio databases), and high concurrent bandwidth. By adopting the method of dynamically adjusting the sampling frequency of the probe points according to the link load, the performance overhead and the accuracy of fault diagnosis are effectively balanced.

[0173] Figure 9 It is a timing diagram of viewing the current state of the audio data processing module in an embodiment of the present application. Among them, / proc / seacraft / adpuInfo is the storage directory of ADPU information in the virtual file system. cat / proc / seacraft / adpuInfo is a command for connecting files and printing to the standard output. Viewing the current state of the audio data processing module is mainly to dynamically judge the detection of the data processing state during the communication process of the ADPU core, and perform data processing on the ADPU core. For example, abnormal operations in the instruction multiplexing, splitting, and distribution operations, as well as abnormal operations during the multi-chip data interaction through the PCIE, will all affect the reliability of the overall data processing. By adding a service for viewing the current state of the audio data processing module, a real-time state detection mechanism is added to judge whether the processing result is reasonable, and the dynamic comprehensive analysis result is fed back to the system management module to achieve the goal of timely error correction.

[0174] Figure 10It is a timing diagram for verifying the encryption and decryption results of an embodiment of the present application. Among them, DMA represents Direct Memory Access. In a multi-core heterogeneous system, data transmission between different modules can be ensured to be secure through dynamic encryption and decryption, and the privacy and security of users can be guaranteed.

[0175] Figure 11 It is a timing diagram for the heartbeat mechanism of driver and firmware communication in an embodiment of the present application. The heartbeat mechanism of driver and firmware communication mainly solves the problems of abnormal states during inter-core communication, abnormal loss and incompleteness of bus data, and dynamic calculation of compensation factor parameters. By means of settable dynamic keep-alive parameters, the running state of communication between multiple cores can be obtained, so as to perform dynamic adjustment to ensure the reliability and secure domain and level control between the overall multi-core systems.

[0176] An embodiment of the present application also provides a device control device based on multi-modal multi-core heterogeneity, including a processor; a memory, in which executable instructions of the processor are stored; wherein, the processor is configured to execute the steps of the device control method based on multi-modal multi-core heterogeneity as described above by executing the executable instructions.

[0177] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method, or a computer program product. Therefore, various aspects of the present application can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "unit" or "platform" here.

[0178] Next, refer to Figure 12 to describe the electronic device 600 according to this embodiment of the present application. Figure 12 The displayed electronic device 600 is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0179] As Figure 12 shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0180] Among them, the storage unit stores program codes, and the program codes can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present application described in the above part of the device control method based on multi-modal multi-core heterogeneity of this specification. For example, the processing unit 610 can execute asFigure 1 The steps shown in

[0181] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only storage unit (ROM) 6203.

[0182] The storage unit 620 may also include a program / utilities 6204 having a set (at least one) of program units 6205. Such program units 6205 include, but are not limited to: an operating system, one or more application programs, other program units, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0183] The bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0184] The electronic device 600 may also communicate with one or more external devices 700 (such as a keyboard, a pointing device, a Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or may communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be carried out through an input / output (I / O) interface 650. Moreover, the electronic device 600 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other units of the electronic device 600 through the bus 630. It should be understood that, although not shown in the figure, other hardware and / or software units may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0185] In the device control device based on multi-modal multi-core heterogeneity, when the program in the memory is executed by the processor, the steps of the device control method based on multi-modal multi-core heterogeneity are implemented. Therefore, the device can also obtain the technical effects of the above-mentioned device control method based on multi-modal multi-core heterogeneity.

[0186] An exemplary embodiment of the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned device control method based on multi-modal multi-core heterogeneity are implemented.

[0187] In one embodiment, a computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium may be a storage medium based on signals such as electricity, magnetism, light, electromagnetic, infrared, etc., including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, mechanical hard disk (HDD), solid state drive (SSD), and so on. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing the computer program, such as read-only memory, Nand Flash, etc.

[0188] In one embodiment, a computer program product may be an intangible product containing a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as an executable file storing the computer program, a digital file such as an installation package.

[0189] The code of the computer program can be written in one or more programming languages. Programming languages such as C, Java, C++, Python, etc. The program code can be executed entirely on the user's computing device, or partially on the user's computing device, or executed as an independent software package, or partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (for example, through an Internet connection provided by an operator).

[0190] The computer program can be carried or transmitted by signals such as electricity, magnetism, light, electromagnetic, infrared, etc. The electronic device can convert the signal carrying the computer program into a digital signal and then run the computer program. When the computer program runs on the electronic device, its code is used to cause the electronic device to execute (more specifically, can cause the processor of the electronic device to execute) the method steps of various exemplary embodiments of the present application, such as can execute the steps of the above-mentioned device control method based on multi-modal multi-core heterogeneity.

[0191] When the computer program is executed by the processor, it implements the steps of the above-mentioned device control method based on multi-modal multi-core heterogeneity. Therefore, the computer program product can also obtain the technical effects of the above-mentioned device control method based on multi-modal multi-core heterogeneity.

[0192] The above content is a further detailed description of the present application in combination with specific preferred embodiments. It cannot be determined that the specific implementation of the present application is only limited to these descriptions. For those of ordinary skill in the technical field to which the present application pertains, without departing from the concept of the present application, several simple deductions or substitutions can be made, and all should be regarded as falling within the protection scope of the present application.

Claims

1. A device control method based on multi-mode multi-core heterogeneity, characterized in that: Based on the control module implementation, the method includes the following steps: Acquire the data to be transmitted from the multimodal media data acquisition module and store it in the data buffer; At every first preset time, the soft interrupt unit determines the backlog state of the data transmission task in the data buffer. If a backlog occurs, all the data to be transmitted in the data buffer are packaged and stored in the designated storage area. If no backlog occurs, the data to be transmitted in the data buffer are sequentially read and stored in the designated storage area. At every second preset time, the hard interrupt unit reads information to be transmitted from the designated storage area, the information to be transmitted including data or data packets to be transmitted, and sends the information to the media data processing module, the media data processing module and the control module have different processing cores, and the media data processing module is configured to obtain a control instruction analysis result based on the received data or data packets; The control instruction is sent to the controlled device based on the control instruction analysis result determined by the media data processing module.

2. The device control method based on multi-mode multi-core heterogeneity according to claim 1, characterized in that: The information to be transmitted also includes a command to be transmitted, and the method further includes the following steps: Acquire commands to be transmitted and store them in a command buffer, wherein if the command transmission tasks in the command buffer are backlogged and the number of current command transmission tasks does not exceed the maximum command number threshold, the command buffer is capacity expanded; if the number of command transmission tasks in the command buffer exceeds the maximum command number threshold, the capacity is not expanded and an exception notification is issued; At every first preset time, the soft interrupt unit reads the to-be-transmitted command from the command buffer and stores the command in the designated storage area.

3. The device control method based on multi-mode multi-core heterogeneity according to claim 1, characterized in that: The soft interrupt unit is further used to read the command to be transmitted from the command buffer and store it in a designated storage area, wherein the information to be transmitted also includes the command to be transmitted; The hard interrupt unit reads the information to be transmitted from the designated storage area, comprising the following steps: The hard interrupt unit determines the priority of the information to be transmitted in the designated storage area; The hard interrupt unit reads the information to be transmitted in the designated storage area in order of priority and sends it to the media data processing module; wherein the hard interrupt unit reads the information to be transmitted whose priority is higher than a preset level threshold and sends it to the media data processing module in sequence, and packages the information to be transmitted whose priority is lower than or equal to the preset level threshold and sends it to the media data processing module.

4. The device control method based on multi-mode multi-core heterogeneity according to claim 1, characterized in that: Pre-constructing a general error code lookup table for the control module and the media data processing module, and pre-constructing a policy lookup table corresponding to a general error code and an error handling policy; The method further comprises the steps of: receiving abnormal information from the control module or the media data processing module; Determine the general error code corresponding to the abnormal information based on the general error code lookup table; Based on the strategy lookup table, searching for an error handling strategy corresponding to a determined universal error code; Determine whether the number of executions of the error handling strategy within a preset time period exceeds a preset number threshold; If yes, reset the hardware device corresponding to the module that sends the abnormal information; If not, the error handling strategy is executed.

5. The device control method based on multi-mode multi-core heterogeneity according to claim 1, characterized in that: The control module communicates with the plurality of media data processing modules respectively through PCIE links; the method further comprises the following steps: Monitor the load status of each PCIE link, and determine the point collection frequency of the media data processing module corresponding to the PCIE link according to the load status; Based on the embedding point collection frequency of each of the media data processing modules, the embedding point data is obtained from the media data processing modules through the PCIE link.

6. The device control method based on multi-mode multi-core heterogeneity according to claim 1, characterized in that: The multimodal media data acquisition module includes a plurality of video acquisition devices; the media data processing module is configured to obtain the control instruction analysis result by adopting the following steps: Extract features from video data of a single video acquisition device respectively; Input the features of each video acquisition device into the gesture classification model respectively to obtain the gesture category corresponding to a single video acquisition device and the confidence level of the gesture category; The gesture categories and confidence levels corresponding to the various video acquisition devices are used as control instruction analysis results and sent to the control module; The control module sends the control instruction to the controlled device based on the control instruction analysis result determined by the media data processing module by the following steps: Receiving the control instruction analysis result, and using the confidence as the weight of the corresponding video acquisition device; Determining whether a gesture is detected according to the gesture category; If the number of gestures detected within a preset time period is greater than a preset gesture number threshold, the weights of the respective video acquisition devices are adjusted to suppress a maximum value in the weights; The classification results of the multiple video acquisition devices are weighted averaged to determine an average gesture category, and a control instruction is generated according to the average gesture category and sent to the controlled device.

7. The device control method based on multi-mode multi-core heterogeneity according to claim 1, characterized in that: The multimodal media data acquisition module includes a plurality of video acquisition devices; the media data processing module is configured to obtain the control instruction analysis result by adopting the following steps: Extract features from video data of a single video acquisition device respectively; Clustering the features of multiple video acquisition devices, selecting a group of features with the most clustered features to integrate to obtain input features, inputting the input features into a gesture classification model to obtain a gesture category with the highest confidence, and using the gesture category as a control instruction analysis result; The control module receives the control instruction analysis result, generates a control instruction according to the gesture category, and sends the control instruction to the controlled device.

8. A device control system based on multi-modal multi-core heterogeneity, characterized in that: include: A control module, used to implement the device control method based on multi-modal multi-core heterogeneity according to any one of claims 1 to 7; The media data processing module has a different processing core from the control module and is used to generate a control instruction for controlling the controlled device based on the received data or data packet.

9. A device control device based on multi-mode multi-core heterogeneity, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the device control method based on multi-modal multi-core heterogeneity according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the device control method based on multi-modal multi-core heterogeneity according to any one of claims 1 to 7 are implemented.

Images