Cooperative control system based on edge computing speech recognition

Abstract

The invention discloses a cooperative control system based on edge computing speech recognition. The cooperative control system comprises speech acquisition equipment and an edge server, the voice acquisition equipment comprises a microphone array, an audio processing module and a communication module which are respectively used for acquiring voice to obtain an audio signal, preprocessing the audio signal to obtain an audio stream and transmitting the audio stream; the edge server is in communication connection with the voice acquisition equipment and the collaborative operation equipment, receives audio streams and controls the collaborative operation equipment to operate, and comprises a main processor and a coprocessor which are respectively a CPU (Central Processing Unit) processor and an NPU (Network Processing Unit) processor; the coprocessor is used for performing parallel processing on automatic voice recognition and text conversion voice; the automatic speech recognition obtains a natural language from an audio stream; the text-to-voice conversion means that the broadcast text is converted into broadcast audio; the main processor is used for processing natural languages and outputting control instructions and / or broadcast texts according to the natural languages; the control instruction is used for controlling the cooperative operation equipment to operate. The processing efficiency is improved, and the delay time is shortened.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and in particular relates to a collaborative control system based on edge computing voice recognition. BACKGROUND

[0002] With the development of smart home scenes, intelligent collaboration is increasingly using voice centralized control to achieve the collaboration of multiple collaborative operations. For example, the voice control of the cooking equipment is started, and the exhaust hood is started to operate at the same time, improving the use experience.

[0003] Currently, voice control is generally divided into two schemes: one is an online voice control scheme, and the other is a local voice control scheme. The online voice control scheme uses a local device to send a user's voice instruction to a cloud server to identify and process the voice instruction through the cloud server, and then returns control information corresponding to the voice instruction to the local device. In this scheme, since the cloud server is needed to identify the voice, the communication status between the local device and the cloud server directly affects the effectiveness of voice control. The local voice control scheme directly adds a voice semantic processing control board to each intelligent collaborative operation device to identify and process the voice instruction through the local voice semantic processing control board, resulting in high costs for the entire house, and poor synchronization and poor collaborative performance of only a single appliance when starting in the presence of faulty appliances.

[0004] The above information disclosed in the background of the application is only used to increase the understanding of the background of the application, and therefore, it can include prior art known to those skilled in the art. SUMMARY

[0005] The present application is directed to the problem of high cost caused by the separate setting of voice semantic processing control boards in the local voice control scheme of the prior art, and proposes a collaborative control system based on edge computing voice recognition, which uses multiple collaborative operation devices to share a local edge distributed server for voice recognition, thereby reducing costs. And by using parallel processing, the voice recognition efficiency is improved, and the control efficiency is further improved.

[0006] To achieve the above-mentioned application / design purposes, the present application adopts the following technical solutions: A collaborative control system based on edge computing voice recognition, comprising a voice collection device and an edge server. The voice collection device comprises a microphone array, an audio processing module and a communication module, respectively used for voice collection to obtain an audio signal, preprocessing of the audio signal to obtain an audio stream, and transmission of the audio stream. The edge server is in communication connection with the voice collection device and multiple collaborative operation devices, receives the audio stream and controls operation of each of the collaborative operation devices, and includes a main processor and a coprocessor; the main processor and the coprocessor are respectively a CPU-type processor and an NPU-type processor; the coprocessor is used for parallel processing of automatic speech recognition and text-to-speech conversion; the automatic speech recognition is to obtain natural language from the audio stream; the text-to-speech conversion is to convert broadcast text into broadcast audio; the main processor is used for natural language processing, and outputs a control instruction and / or the broadcast text according to the natural language; the control instruction is used for controlling operation of each of the collaborative operation devices.

[0007] In some specific embodiments, the voice collection device adopts an ESP32 end-side device configured in a deep sleep wake-up mode, which includes: setting a frequency threshold and an energy threshold; noise reduction and frame acquisition of the audio signal to obtain the audio stream; acquiring multiple audio frames of the audio stream; and performing spectrum analysis on the audio frames; judging whether the frequency of the audio frame is within the frequency threshold range; if yes, calculating the energy value of each of the audio frames; judging whether the energy values of the continuous multiple audio frames are within the energy threshold range; if yes, starting frame recording; if no, continuing to sleep.

[0008] In some specific embodiments, the main processor controls the collaborative operation of the collaborative operation device by using a dynamic process table, including: analyzing the natural language to generate an initial task sequence; writing the initial task sequence into a dynamic process table; a task scheduling engine generates an execution sequence according to the dynamic process table, and generates a device instruction according to the execution sequence; receiving device state feedback and performing feedback state judgment; if normal, marking the task as completed; if abnormal, reorganizing the dynamic process table, updating the execution sequence by the task scheduling engine, and generating the device instruction; when the task is completed, judging whether there is a subsequent task; if yes, the task scheduler continues to execute; if no, completing.

[0009] In some specific embodiments, the main processor controls each of the collaborative operation devices by using a dynamic process table, further including: The task scheduling engine first determines whether the execution priority is met when executing the sequence according to the dynamic process table; if yes, the device instruction is issued; if no, the dynamic process table is reorganized, the execution sequence is updated by the task scheduling engine, and the device instruction is issued.

[0010] In some specific embodiments, the data structure of the dynamic process table includes task identification, target device identification, device state, target state, pre-task identification, timeout time, priority, and execution state; the execution state includes pending, executing, completed, and failed. The dynamic process table dynamically manages the tasks through the device state, including: The device state includes ACTIVE, ERROR, and STANDBY. The task scheduling engine only processes the cooperative operation device nodes with the device state of ACTIVE; when the task of the cooperative operation device node fails abnormally, the cooperative operation device node is marked as ERROR, removed from the current execution sequence, and all the in-degree of its successor task nodes is rechecked; when the standby device is enabled, a new node marked as ACTIVE is dynamically injected, the task scheduling engine integrates it into the current execution sequence, and recalculates the dependency relationship.

[0011] In some specific embodiments, the communication data frame between the voice collection device, the edge server, and the cooperative operation includes a frame header, a timestamp, a data type, a data length, a payload, and a check. The timestamp is in microseconds; the data type includes voice stream, control instruction, and device state feedback; a uniform execution time is set in the device instruction of each cooperative operation device.

[0012] In some specific embodiments, the main processor generates and executes the execution sequence by using the Kahn algorithm integrated with the priority queue, including: Find all task nodes with an in-degree of 0; put them into a queue based on task priority; Execute the task node with the highest priority from the queue each time.

[0013] In some specific embodiments, the automatic speech recognition and the text-to-speech model share a memory space, different virtual addresses are allocated to the automatic speech recognition and the text-to-speech model processes by a memory management unit (MMU) or a memory protection unit (MPU), which are mapped to the same physical address; the memory space is controlled by authority according to the physical address segmentation or paging; the authority includes read-only, executable, and read-write.

[0014] In some specific embodiments, the memory space is fragmented based on the operator dependency relationship and historical execution flow of the automatic speech recognition and the text-to-speech model; a group of operators with strong continuity and high dependency is divided into a fragment; the automatic speech recognition and the text-to-speech model are loaded in a fragmented manner, and the active fragment is loaded first; According to the current executed fragment, the most likely required fragment is predicted, and it is loaded in the background in advance.

[0015] In some specific embodiments, the memory fragmentation of the co-processor includes: Static fragmentation, which is to set a static fragmentation threshold, and when the fragmentation rate of the memory reaches the static fragmentation threshold, the fragmentation is triggered; Incremental fragmentation, which is to allocate memory fragmentation to multiple time slices; Idle period fragmentation, which is a preventive fragmentation actively performed during idle or low load without judging the fragmentation static threshold.

[0016] Compared with the prior art, the advantages and positive effects of the present application are: The edge computing-based voice recognition collaborative control system of the present application realizes local voice recognition and controls each collaborative operation device according to the recognized voice by setting an edge server connected with multiple collaborative operation devices and a voice collection device connected with the edge server, thereby avoiding the privacy risks, network congestion, low efficiency, and network failure of network server voice recognition, shortening the control delay time, and improving the efficiency, reliability, and privacy of voice control collaborative operation. In addition, the edge server includes a CPU processor-based main processor and an NPU processor-based co-processor, so that voice recognition and voice generation, logical processing are performed by the co-processor of the NPU and the main processor of the CPU, respectively, and the co-processor can process voice recognition and voice generation in parallel, thereby taking advantage of different types of processors, improving processing efficiency, further shortening control delay time, and improving user experience.

[0017] Other features and advantages of the present application will become more apparent after reading the specific embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0019] Figure 1 is a schematic diagram of the composition architecture of the collaborative control system based on edge computing voice recognition according to the embodiments; Figure 2 is a schematic diagram of the wake-up process of the voice collection device according to the embodiments; Figure 3 is a schematic diagram of the process controlled by the dynamic process table according to the embodiments; Figure 4 is a schematic diagram of the composition of the dynamic process table data structure according to the embodiments; Figure 5 is a schematic diagram of the composition of the communication frame according to the embodiments; Figure 6 is a schematic diagram of the control timing of the collaborative control system based on edge computing voice recognition according to the embodiments; Figure 7 is a schematic diagram of the function distribution of the edge server according to the embodiments; Figure 8 is a running flowchart when the cloud server is configured according to the embodiments; Figure 9 is a running flowchart when the cloud server is configured according to the embodiments. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the scope of protection of the present application.

[0021] In the description of the present application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore, it cannot be understood as a limitation of the present application.

[0022] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, can be fixedly connected, can be detachably connected, or integrally connected. For those skilled in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In the description of the embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0023] The terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features.

[0024] In the description of the present application, unless otherwise specified, the meaning of "multiple" is two or more.

[0025] Referring to Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , the present application discloses a collaborative control system based on edge computing voice recognition, which comprises a voice collection device and an edge server in communication connection with the voice collection device.

[0026] The voice collection device comprises a microphone array, an audio processing module and a communication module, which are respectively used for voice collection to obtain an audio signal, pre-processing of the audio signal to obtain an audio stream, and transmission of the audio stream to the edge server.

[0027] In addition to the communication connection with the voice collection device, the edge server is also in communication connection with a plurality of collaborative operation devices, receives the audio stream transmitted by the voice collection device, and generates control instructions for controlling the operation of each collaborative operation device according to the audio stream and transmits the control instructions to each collaborative operation device, for controlling the collaborative operation of each collaborative operation device.

[0028] The edge server comprises a main processor and a coprocessor, which respectively adopt a CPU type processor and an NPU type processor; the coprocessor is used for parallel running of an automatic speech recognition model and a text-to-speech model, for performing speech recognition and conversion between text and speech; the automatic speech recognition model is used for analyzing and recognizing the audio stream to obtain natural language; the text-to-speech model is used for converting the broadcast text into broadcast audio; the main processor is used for natural language processing, for obtaining control instructions and / or broadcast text output according to the execution task input by the natural language, for controlling the operation of each collaborative operation device and broadcasting the operation state of each collaborative operation device; the control instructions are used for controlling the operation of each collaborative operation device.

[0029] The collaborative control system based on edge computing voice recognition of the present application realizes local voice recognition and controls each collaborative operation device according to the recognized voice by setting an edge server connected with multiple collaborative operation devices and a voice collection device connected with the edge server, avoids the privacy hidden danger of network server voice recognition and the problem of long delay time and network failure caused by low network blocking efficiency, shortens the control delay time, and improves the efficiency, reliability and privacy of voice control collaborative operation; in addition, the edge server includes a CPU processor type main processor and an NPU processor type coprocessor, so that voice recognition and voice generation, logical processing are respectively performed by the coprocessor of the NPU and the main processor of the CPU, and the coprocessor can process voice recognition and voice generation in parallel, so as to give full play to the advantages of different types of processors, improve the processing efficiency, further shorten the control delay time, and improve the user experience.

[0030] The specific composition, control process and principle of the collaborative operation collaborative control system based on edge computing voice recognition of the present application will be described in detail below through specific embodiments.

[0031] In some specific embodiments, voice recognition and voice generation, logical processing are respectively performed by the coprocessor of the NPU and the main processor of the CPU using a preemption scheduling algorithm based on a task queue.

[0032] In some specific embodiments, with reference to Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 、 Figure 7 The voice collection device uses an ESP end side device, specifically an ESP32 end side device, which integrates WI-FI and Bluetooth communication functions, and further includes a microphone array, an audio processing module, a hardware configuration of a 4-microphone beamforming array (signal-to-noise ratio > 65 dB), an I2S audio interface (sampling rate 16 kHz / bit depth 16 bit), which are respectively used for voice collection and noise reduction and framing of the collected voice; the ESP end side device is configured to operate in a deep sleep wake-up mode, in which the CPU, most of the RAM and digital peripherals (such as Wi-Fi and Bluetooth) are completely powered off, and only the RTC controller, the ULP coprocessor and the RTC memory remain running, which are awakened by the ULP coprocessor, including: S11, setting a frequency threshold and an energy threshold; S12, noise reduction and framing of the audio signal, i.e. voice, to obtain an audio stream, which includes multiple audio frames; S13, obtaining multiple audio frames of the audio stream; performing spectrum analysis on each audio frame; S14, judge whether the frequency of each audio frame is in the range of the frequency threshold; if yes, execute S15; if no, execute S17; S15, calculate the energy value of each audio frame; judge whether the energy values of the continuous multiple audio frames are in the range of the energy threshold; if yes, execute S16; if no, execute S17; S16, start the frame recording; S17, continue to sleep.

[0033] The frame recording in S16 is a technology of dividing the continuous audio signal into short time frames for processing, i.e. starting the transmission of the audio stream signal.

[0034] The voice collection device of the collaborative control system based on the edge computing voice recognition of the embodiment judges whether it is a control voice through the dual judgment of the frequency and the energy, is used for the wake-up start and the sleep control of the device recording algorithm, the frequency judgment avoids the environmental noise false triggering, the energy judgment avoids the peripheral environmental voice false wake-up, and the accuracy and the reliability of the wake-up are improved.

[0035] In some specific embodiments, in S11, the frequency threshold is 300Hz-3400Hz; in S14, judging whether the frequency of each audio frame is in the range of the frequency threshold is judging whether the main frequency of the audio frame is in the range of 300Hz-3400Hz.

[0036] In S15, the judgment of the energy of the multiple audio frames adopts the judgment of whether the energy values of the continuous three frames are in the range of the energy threshold.

[0037] The collaborative control system based on the edge computing voice recognition of the embodiment sets the frequency threshold range of the voice, limits the voice signal in the human voice frequency range, prevents the environmental noise interference, improves the accuracy and the reliability of the judgment, and further improves the accuracy and the reliability of the sleep and the wake-up control.

[0038] In some specific embodiments, with reference to Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , the main processor controls the collaborative operation device by using the dynamic process table, including: S21, analyze the natural language; S22, judge whether to generate an initial task sequence; if yes, execute S23; S23, write the initial task sequence into the dynamic process table; S24, the task scheduling engine generates an execution sequence according to the dynamic process table; S25, issuing device instructions according to the execution sequence; S26, receiving device state feedback; S27, judging the feedback state; if normal, executing S28; if abnormal, executing S29; S28, marking the task as completed; judging whether there is a subsequent task; if yes, executing S25; if no, ending; S29, then reorganizing the dynamic process table and returning to S24.

[0039] In S29, the reorganization of the dynamic process table includes deep traversal of the dependency tree, removal of the abnormal task node and all its successor task nodes, prevention of additional operation of other collaborative operation devices operating in collaboration, reduction of energy waste, and improvement of user experience.

[0040] In an example, referring to Figure 6 , the edge server receives an audio stream of a voice collection device, performs natural language recognition on the audio stream, generates a process table according to the analysis of the natural language, and generates control instructions according to the process table and issues the control instructions to the device group; the device group feeds back the state after receiving the control instructions; the edge server receives and judges the feedback state; when abnormal, a new execution sequence is implemented by reorganizing the process table, and an abnormal notification voice is generated and sent to the voice collection device for broadcasting.

[0041] In some specific embodiments, referring to Figure 7 , the audio stream is transmitted to the edge server, and the preprocessing of the data such as audio framing and normalization is responsible by the main processor, while the voice frame is identified by the ASR neural network inference and is responsible by the coprocessor, and the obtained natural language text is sent to the main processor; the main processor encodes and analyzes the prosody of the natural language text, and transmits the control instructions and / or text broadcast information to each collaborative operation device and / or coprocessor; the coprocessor obtains the broadcast voice by TTS neural network inference and transmits it to the main processor.

[0042] In S29, the reorganization of the dynamic process table includes deep traversal of the dependency tree, removal of the abnormal task node and all its successor task nodes, prevention of additional operation of other collaborative operation devices operating in collaboration, reduction of energy waste, and improvement of user experience.

[0043] In some specific embodiments, referring to Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , the main processor controls the collaborative operation of the collaborative operation device by using a dynamic process table, and further comprises: The task scheduling engine generates an execution sequence according to the dynamic process table, judges whether the execution sequence conforms to the execution according to the priority before issuing the device instruction according to the execution sequence; if yes, S25 is executed to issue the device instruction; if no, S29 is executed to reorganize the dynamic process table, and the execution sequence is updated by the task scheduling engine according to the dynamic process table in S24.

[0044] The collaborative control system based on edge computing voice recognition of the embodiment realizes the priority processing of the emergency exception task by upgrading the priority of the corresponding exception processing task when the collaborative operation device is in an abnormal state, and improves the safety and reliability of the collaborative control operation in the manner of real-time monitoring of the device state, management of the task dependency relationship, and dynamic adjustment of the dynamic process table of the execution sequence.

[0045] In some specific embodiments, with reference to Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , the Kahn algorithm with priority queue is used to generate and issue the device instruction of the execution sequence by the main processor, including: S31, find all task nodes with an in-degree of 0; put them into a queue based on task priority; S32, execute the task node with the highest priority in the queue each time.

[0046] The collaborative control system based on edge computing voice recognition of the embodiment improves the safety of operation by setting the highest priority for priority processing when an emergency occurs by integrating the priority into the task scheduling engine.

[0047] In some specific embodiments, the standard JSON is used to set a unified intermediate layer protocol in the edge server, and various communication protocols such as Zigbee / BLE / Matter are converted to the standard JSON protocol to realize the non-inductive compatible collaborative control of the multi-brand collaborative operation device.

[0048] In some specific embodiments, with reference to Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , the data structure of the dynamic process table includes task identification, target device identification, device state, target state, pre-task identification, timeout time, priority, and execution state; the execution state includes pending, executing, completed, and failed; The main processor realizes dynamic management of the tasks of the collaborative operation devices through monitoring of the device states, i.e., dynamic process table management, including: S41, the preset device states include ACTIVE, ERROR and STANDBY. S42, when the task of a collaborative operation device node fails abnormally, the collaborative operation device node is marked as ERROR, and is removed from the current execution sequence, and the in-degree of all successor task nodes thereof is rechecked; when a standby device is enabled, a new node marked as ACTIVE is dynamically injected, and the task scheduling engine integrates the new node into the current execution sequence, and recalculates the dependency relationship.

[0049] The collaborative control system based on edge computing voice recognition of the embodiment realizes fuse protection of a fault collaborative operation device and seamless connection of a standby collaborative operation device through the task scheduling engine which only processes the collaborative operation device nodes with the device state of ACTIVE and updates the execution sequence according to the device states, thereby improving efficiency and use experience.

[0050] In some specific embodiments, with reference to Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , the communication between the voice collection device, the edge server and the collaborative operation devices adopts a custom binary protocol format of MQTT, and a data frame includes a frame header, a timestamp, a data type, a data length, a payload and a check.

[0051] The data type includes a voice stream, a control instruction and a device state feedback, and is used for marking the types of different communication data.

[0052] The timestamp adopts a microsecond level, and is realized through configuration of a high-precision timer, a high-speed clock source and synchronization to a unified clock source through a PTP protocol. A unified execution time is set in a device instruction of each collaborative operation device, and is used for realizing synchronous execution of each device.

[0053] The collaborative control system based on edge computing voice recognition of the embodiment realizes starting of each collaborative operation device at the same time through the microsecond-level timestamp and setting of the unified execution time in the device instruction, improves the uniformity of operation of each collaborative operation device, and improves use experience of a user.

[0054] In some specific embodiments, with reference to Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 ,Figure 7 The model sharing memory space for automatic speech recognition and text-to-speech is shared by the automatic speech recognition and text-to-speech model processes through a memory management unit (MMU) or a memory protection unit (MPU) to allocate different virtual addresses to the automatic speech recognition and text-to-speech model processes, which are mapped to the same physical address and use the same memory; the memory space is controlled by authority according to the physical address segmentation or paging; the authority includes read-only, executable, and read-write. For example, the memory physical address that can be accessed simultaneously is set to read-only or executable; the memory physical address set to read-write can only be accessed in time-sharing mode, thereby realizing 0-copy shared memory.

[0055] In some specific embodiments, the NPU parallel processing of ASR and TTS reduces the amount of calculation and memory occupation through model quantization and operator fusion, so that the NPU can more efficiently process more tasks at the same time.

[0056] In some specific embodiments, the memory space is divided into slices based on the operator dependency and historical execution flow of the automatic speech recognition and text-to-speech model; an operator group with strong continuity and high dependency is divided into a slice; the automatic speech recognition and text-to-speech model is loaded in a sliced manner, and the active slice is loaded first; then, according to the currently executed slice, the most likely needed slice in the future is predicted and loaded in the background in advance.

[0057] The collaborative control system for edge computing-based speech recognition of the present embodiment improves the memory utilization rate and execution efficiency through the execution of memory slicing, sliced startup of part of the active slices, and preloading of subsequent slices.

[0058] In some specific embodiments, the memory fragmentation arrangement by the coprocessor includes: Static arrangement, which is to set a static fragmentation threshold, and to trigger fragmentation arrangement when the fragmentation rate of the memory reaches the static fragmentation threshold; Incremental arrangement, which is to arrange the memory fragmentation into multiple time slices; a time slice (timeslice) is also called a "quantum" or a "processor slice", which is a micro amount of CPU time allocated by a time-sharing operating system to each running process (in a preemptive kernel: the time from the start of the process to the preemption). Idle period arrangement, which is a preventive arrangement actively performed during idle or low load without judging the static fragmentation threshold.

[0059] The collaborative control system for edge computing-based speech recognition of the present embodiment solves the problem of long memory arrangement time affecting the current calculation task and calculation efficiency by comprehensively using the static arrangement, incremental arrangement, and idle period arrangement methods, thereby improving the calculation efficiency and further improving the control efficiency.

[0060] In some specific embodiments, referring to Figure 1 、 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 , the edge server is connected with the cloud server; the edge server judges the networking condition of the edge server and the cloud server; when the connection and communication of the edge server and the cloud server are normal, that is, networking, the cloud enhanced ASR is used for voice recognition, and the recognized natural language is used to generate control instructions through the NLP of the cloud to improve the accuracy and accuracy of semantic understanding; when the edge server judges that the connection and / or communication of the edge server and the cloud server is abnormal, that is, off-line, the basic version ASR of the edge server is used for voice recognition and the local simplified NLP generates control instructions to control the cooperative operation equipment to ensure that the cooperative operation voice control is available when the cooperative operation equipment is off-line.

[0061] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, the technical solutions recorded in the foregoing embodiments can still be modified or some technical features can be replaced by equivalents for those skilled in the art; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions claimed by the present application.

Claims

1. A collaborative control system based on edge computing and speech recognition, characterized in that, include: A voice acquisition device includes a microphone array, an audio processing module, and a communication module, which are respectively used for voice acquisition to obtain audio signals, preprocessing the audio signals to obtain an audio stream, and transmitting the audio stream; An edge server, communicatively connected to the voice acquisition device and multiple collaborative operating devices, receives the audio stream and controls the operation of each collaborative operating device, including a main processor and a coprocessor; the main processor and the coprocessor are respectively CPU-type processors and NPU-type processors; the coprocessor is used for parallel processing of automatic speech recognition and text-to-speech conversion; the automatic speech recognition is to obtain natural language from the audio stream; the text-to-speech conversion is to convert the broadcast text into broadcast audio; the main processor is used for natural language processing, to output control commands and / or the broadcast text according to the natural language; the control commands are used to control the operation of each collaborative operating device.

2. The collaborative control system based on edge computing speech recognition according to claim 1, characterized in that, The voice acquisition device uses an ESP32 edge device, configured in deep sleep wake-up mode, and includes: Set frequency threshold and energy threshold; The audio signal is denoised and the audio stream is obtained by frame segmentation; Acquire multiple audio frames from the audio stream; perform spectral analysis on the audio frames; Determine whether the frequency of the audio frame is within the frequency threshold range; if so, calculate the energy value of each audio frame. Determine whether the energy values ​​of multiple consecutive audio frames are within the energy threshold range; if yes, start frame-by-frame recording; if no, continue to sleep.

3. The collaborative control system based on edge computing speech recognition according to claim 1, characterized in that, The main processor uses a dynamic process table to control the collaborative operation, including: The natural language is parsed to generate an initial task sequence; Write the initial task sequence into the dynamic process table; The task scheduling engine generates an execution sequence according to the dynamic process table, and generates and issues device instructions according to the execution sequence; Receive device status feedback and determine the feedback status; if normal, mark the task as completed; if abnormal, reorganize the dynamic process table, and have the task scheduling engine update the execution sequence and generate and issue the device instructions. When a task is completed, it is determined whether there are any follow-up tasks; if so, the task scheduler continues execution; otherwise, the task is completed.

4. The collaborative control system based on edge computing speech recognition according to claim 3, characterized in that, The main processor uses a dynamic process table to control each of the collaboratively operating devices, and also includes: When the task scheduling engine executes the execution sequence according to the dynamic process table, it first determines whether the execution priority is met; if yes, it issues the device instruction; if no, it reorganizes the dynamic process table, and the task scheduling engine updates the execution sequence and generates and issues the device instruction.

5. The collaborative control system based on edge computing speech recognition according to claim 3 or 4, characterized in that, The data structure of the dynamic process table includes task identifier, target device identifier, device status, target status, preceding task identifier, timeout, priority, and execution status; the execution status includes pending, executing, completed, and failed. The dynamic process table dynamically manages the tasks based on the device status, including: The device status includes ACTIVE, ERROR, and STANDBY; The task scheduling engine only processes the collaboratively operating device nodes whose device status is ACTIVE. When the task of the collaboratively operating device node fails due to an anomaly, the collaboratively operating device node is marked as ERROR and removed from the current execution sequence. The in-degree of all its successor task nodes is then rechecked. When a backup device is enabled, a new node marked as ACTIVE is dynamically injected. The task scheduling engine integrates this node into the current execution sequence and recalculates the dependencies.

6. The collaborative control system based on edge computing speech recognition according to claim 5, characterized in that, The communication data frames between the voice acquisition device, the edge server, and the collaborative operation include a frame header, timestamp, data type, data length, payload, and verification. The timestamps are in microseconds; the data types include voice streams, control commands, and device status feedback; and the device commands of each of the collaboratively operating devices have a unified execution time.

7. The collaborative control system based on edge computing speech recognition according to claim 5, characterized in that, The main processor uses the Kahn algorithm with priority queues to generate and execute the execution sequence, including: Find all the task nodes with an in-degree of 0; add them to a queue based on task priority; Each time, the task node with the highest current priority is taken from the queue and executed.

8. The collaborative control system based on edge computing speech recognition according to any one of claims 1 to 4, characterized in that, The automatic speech recognition and text-to-speech models share a memory space. Different virtual addresses are allocated to the automatic speech recognition and text-to-speech model processes through a memory management unit (MMU) or a memory protection unit (MPU), which are mapped to the same physical address. The memory space is segmented or paginated according to the physical address for access control. The permissions include read-only, executable, and read-write.

9. The collaborative control system based on edge computing speech recognition according to claim 8, characterized in that, The memory space is partitioned based on the operator dependencies and historical execution pipelines of the automatic speech recognition and text-to-speech models; groups of operators with strong continuity and high dependencies are divided into partitions; the automatic speech recognition and text-to-speech models are loaded using a partitioned loading method, with the active partitions being loaded first; Based on the currently executing shard, predict the next shard that will be most likely to be needed and load it in advance in the background.

10. The collaborative control system based on edge computing speech recognition according to claim 9, characterized in that, The coprocessor's defragmentation of memory includes: Static defragmentation involves setting a static fragmentation threshold and triggering defragmentation when the memory fragmentation rate reaches that threshold. Incremental defragmentation involves spreading memory fragmentation across multiple time slices. Idle period defragmentation is a preventative defragmentation process that is proactively performed during periods of idleness or low load, without determining static fragmentation thresholds.

Images