Intercom method and electronic device applied to a back-of-house environment

Abstract

This disclosure provides a two-way intercom method and electronic device for use in a kitchen environment, relating to the field of Internet technology. The method includes: receiving a message sequence from a kitchen monitoring terminal; wherein the message sequence contains payload data using an MPEG-2 program stream encapsulation format; based on the encapsulation header identifier in the payload data, converting the payload data into a distribution data stream conforming to a streaming media protocol and distributing it to a client, the client being used to play video footage and audio from the kitchen environment; acquiring an intercom audio stream for the monitored content in the payload data; wherein the encoding format of the intercom audio stream matches the audio encoding format in the payload data; and in response to an intercom command sent by the client, pushing the intercom audio stream to the kitchen monitoring terminal through a voice feedback link established with the kitchen monitoring terminal. This disclosure embodiment achieves two-way real-time intercom functionality between the client and the kitchen monitoring terminal while maintaining compatibility with existing national standard equipment.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, and in particular to a two-way communication method and electronic device for use in a kitchen environment. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] With the continuous deepening of food safety supervision and the general improvement of public awareness of food safety, "transparent kitchens" have become a key part of the standardization of the catering industry, and remote, real-time visual supervision of kitchen operations has become a rigid requirement. Against this backdrop, video surveillance equipment based on national standards has been widely used.

[0004] Therefore, how to achieve efficient interaction between the client and the kitchen based on national standard equipment is the technical problem that this application aims to solve. Summary of the Invention

[0005] The purpose of this disclosure is to provide an intercom method and electronic device for use in a kitchen environment, which can enable efficient interaction between the client and the kitchen monitoring terminal.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] This disclosure provides an intercom method for use in a kitchen environment, comprising: receiving a message sequence from a kitchen monitoring terminal; wherein the message sequence includes payload data encapsulated in MPEG-2 program stream format; converting the payload data into a distribution data stream conforming to a streaming media protocol and distributing it to a client based on the encapsulation header identifier in the payload data, the client being used to play video footage and audio from the kitchen environment; acquiring an intercom audio stream for the monitoring content in the payload data; wherein the encoding format of the intercom audio stream matches the audio encoding format in the payload data; and, in response to an intercom command sent by the client, pushing the intercom audio stream to the kitchen monitoring terminal via a voice feedback link established with the kitchen monitoring terminal.

[0008] In some embodiments, the method further includes: in response to the intercom command, dynamically allocating an idle port from a pre-configured port pool as the server-side listening port of the voice backhaul link; sending the address information of the server-side listening port to the kitchen monitoring terminal, so that the kitchen monitoring terminal can initiate a connection request to the server-side listening port to establish the voice backhaul link.

[0009] In some embodiments, the method further includes: during the process of pushing the intercom audio stream to the kitchen monitoring terminal, detecting real-time audio streams in the monitored content in real time; triggering an interruption event in response to detecting new voice activity from the real-time audio stream; and performing at least one of the following operations in response to the interruption event: interrupting the pushing of the intercom audio stream to the kitchen monitoring terminal; clearing unsent audio frames in the audio buffer; and starting to acquire and push a new intercom audio stream corresponding to the new voice activity.

[0010] In some embodiments, pushing the intercom audio stream to the kitchen monitoring terminal includes: obtaining the current system time and the starting push system time for pushing the intercom audio stream to the kitchen monitoring terminal; determining the difference between the current system time and the starting push system time; determining the audio index of the audio frame that should be sent in the intercom audio stream based on the starting push system time and the difference, as the first audio frame index; extracting the audio frame corresponding to the first audio frame index from the audio buffer and pushing it to the kitchen monitoring terminal.

[0011] In some embodiments, pushing an intercom audio stream to a kitchen monitoring terminal includes: obtaining a first video timestamp corresponding to the monitoring content played by the client when the intercom is triggered, as the start time for the kitchen monitoring terminal to simultaneously play the intercom audio stream while playing the monitoring content; obtaining a second video timestamp corresponding to the monitoring content currently being played by the client; calculating the time offset between the second video timestamp and the first video timestamp; determining a target audio frame corresponding to the time offset from the intercom audio stream based on the time offset, wherein the start time of the intercom audio stream corresponds to the first video timestamp; determining a target audio frame to be sent from the intercom audio stream based on the time offset; extracting the target audio frame from the audio buffer, associating the target audio frame with the second video timestamp, and pushing it to the kitchen monitoring terminal so that the kitchen monitoring terminal synchronously plays the target audio frame when playing the monitoring video screen corresponding to the second video timestamp.

[0012] In some embodiments, acquiring an intercom audio stream for the monitored content in the payload data includes: in response to a voice intercom request initiated by the client, establishing an uplink long connection with the client using the WebSocket long connection protocol; and receiving an intercom audio stream for the monitored content in the payload data from the client through the uplink long connection.

[0013] In some embodiments, the method further includes: extracting audio and converting audio format of the payload data to obtain target audio; performing speech recognition on the target audio to obtain recognized text information; inputting the recognized text information into a large language model so that the large language model generates response text based on the recognized text information; synthesizing a first prompt audio stream based on the response text; and pushing the first prompt audio stream to the kitchen monitoring terminal through the voice feedback link.

[0014] In some embodiments, the method further includes: extracting kitchen business features from the monitored content; performing knowledge matching between the kitchen business features and vector data in a preset kitchen knowledge base, so as to generate an audio stream that conforms to the kitchen business scenario based on the matched preset kitchen knowledge constraint large language model.

[0015] In some embodiments, the method further includes generating a second intercom audio stream by at least one of the following methods, and pushing the second intercom audio stream to the kitchen monitoring terminal via the voice backhaul link: detecting kitchen violations in the monitored content; if a preset violation feature is detected, determining a preset alarm text corresponding to the violation feature; converting the preset alarm text into the second intercom audio stream and pushing it to the kitchen monitoring terminal; acquiring structured kitchen order data from the kitchen business system; parsing the structured kitchen order data based on preset trigger conditions to generate an order prompt text instruction; and inputting the order prompt text instruction into a speech synthesis model to generate the second intercom audio stream.

[0016] This disclosure provides an electronic device comprising: a memory and a processor; the memory for storing computer program instructions; and the processor for calling the computer program instructions stored in the memory to implement the intercom method for use in a kitchen environment as described above.

[0017] This disclosure provides a computer-readable storage medium storing computer program instructions to implement the intercom method applied in a kitchen environment as described in any of the preceding embodiments.

[0018] This disclosure provides a computer program product or computer program that includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and the processor executes the computer program instructions to implement the aforementioned intercom method applied in a kitchen environment.

[0019] The intercom method and electronic device for use in a kitchen environment provided in this disclosure parse the MPEG-2 PS encapsulated stream of the kitchen monitoring terminal and convert it into a distribution data stream of the standard streaming media protocol to achieve cross-platform real-time playback. At the same time, it establishes a voice feedback link with the kitchen monitoring terminal and pushes the intercom audio stream with matching encoding format to the camera, thereby realizing two-way real-time intercom function in the kitchen environment while being compatible with existing national standard equipment.

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 A schematic diagram of a scenario is shown, illustrating an intercom method or intercom device applicable to a kitchen environment that can be applied to embodiments of this disclosure.

[0023] Figure 2 This is a flowchart illustrating an intercom method applied in a kitchen environment according to an exemplary embodiment.

[0024] Figure 3 This is a flowchart illustrating an intercom method applied in a kitchen environment according to an exemplary embodiment.

[0025] Figure 4 This is a schematic diagram of a live streaming link architecture according to an exemplary embodiment.

[0026] Figure 5 This is a diagram illustrating a voice link structure according to an exemplary embodiment.

[0027] Figure 6 This is a flowchart illustrating an intelligent two-way dialogue method according to an exemplary embodiment.

[0028] Figure 7 This is a schematic diagram of an intelligent media processing engine according to an exemplary embodiment.

[0029] Figure 8 This is a flowchart illustrating a voice playback method according to an exemplary embodiment.

[0030] Figure 9This is a flowchart illustrating a dialogue method according to an exemplary embodiment.

[0031] Figure 10 This is a flowchart illustrating a dialogue method according to an exemplary embodiment.

[0032] Figure 11 This is a block diagram illustrating an intercom device applied in a kitchen environment according to an exemplary embodiment.

[0033] Figure 12 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0034] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0035] Those skilled in the art will recognize that embodiments of this disclosure can be a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0036] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0037] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0038] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0040] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the terms "contains," "includes," and "has" are used to indicate an open-ended meaning of inclusion and refer to the existence of additional elements / components / etc. besides those listed.

[0041] This disclosure embodiment can be implemented by a terminal and / or a server. The terminal can obtain data from a computer device and display that data. The computer device can interact with the terminal, and can be a server hosting the application, or it can belong to the terminal (i.e., the terminal's backend), etc., without limitation.

[0042] The terminal can be a mobile phone, a laptop computer, or a playback device in a vehicle, etc., without limitation. The terminal can be considered a playback device in a vehicle, and it can display the target application. The terminal is only one example of the devices listed; the terminal in this disclosure is not limited to the listed devices. The target application in this disclosure can be any application capable of displaying multimedia information.

[0043] It is understood that the terminal mentioned in the embodiments of this disclosure can be a computer device, including but not limited to a terminal or a server. In other words, the computer device can be a server or a terminal, or a system composed of a server and a terminal. The terminal mentioned above can be an electronic device, including but not limited to mobile phones, tablets, desktop computers, laptops, handheld computers, in-vehicle devices, augmented reality / virtual reality (AR / VR) devices, head-mounted displays, smart TVs, wearable devices, smart speakers, digital cameras, webcams, and other mobile internet devices (MIDs) with network access capabilities, or terminals in scenarios such as trains, ships, and flights.

[0044] The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, vehicle-road cooperation, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0045] Optionally, the data involved in the embodiments of this disclosure may be stored in a computer device or may be stored based on cloud storage technology, without limitation.

[0046] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0047] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0048] For example, when monitoring the kitchen environment through the kitchen monitoring terminal, the technical solution of this application strictly complies with relevant national laws and regulations, and obtains explicit authorization and consent from the catering service provider (such as merchants and kitchen managers) before deploying the monitoring equipment, ensuring the legality and transparency of the monitoring behavior. In scenarios involving user voice interaction, this solution can clearly inform users of the purpose, processing method, and storage period of the voice data through the client interface before collecting user voice data, and only starts the collection process after obtaining the user's active consent.

[0049] For example, when disseminating and storing monitored content, this solution implements strict access control and encrypted storage for audio and video data, restricting access only to authorized supervisors, merchant managers, or the user (within their authorized scope). Data transmission utilizes encrypted channels to prevent unauthorized interception or tampering during transmission. For monitoring data that has reached its storage expiration date or is no longer needed, an automated, periodic cleanup mechanism can be established to delete or anonymize it in accordance with laws and regulations, ensuring compliant management of data throughout its lifecycle. Furthermore, multi-layered security measures can be deployed, including tiered access control, operation log retention and auditing, and abnormal access behavior monitoring, to prevent data leakage and unauthorized access risks, comprehensively protecting user personal information security and system network security.

[0050] To facilitate clarity, the following explanation will focus on the specific application scenario of "transparent kitchens" to illustrate the technical problem this application aims to solve. It should be understood that this scenario description is merely illustrative and not restrictive. The method proposed in this application can address the integration and intelligent interaction issues between national standard monitoring protocols and the internet audio / video and AI (Artificial Intelligence) service ecosystem.

[0051] Therefore, the technical solution of this application can be applied to any scenario involving interoperability of heterogeneous protocols and requiring intelligent two-way audio and video communication, such as industrial inspection, remote security command, and smart classrooms.

[0052] With stricter food safety regulations and increased public awareness of food safety, open kitchens have become a standard feature in the food delivery industry.

[0053] In related technologies, real-time live streaming of kitchen videos can be achieved based on national standard protocols (such as GB28181), allowing consumers or food delivery platform operators to watch the kitchen operations through the platform. However, one-way video surveillance can no longer meet the growing demand for interaction.

[0054] In simple terms, food delivery is a business service model where consumers do not go to the store in person, but instead place an order by phone or online platform, and the merchant or third-party service personnel prepare the meal (or goods) and deliver it to the designated location.

[0055] Two-way intercom systems in related technologies can achieve point-to-point voice calls based on the SIP (Session Initiation Protocol) protocol or dedicated intercom equipment, but require manual operation and response, resulting in high labor costs and slow response speed, and cannot achieve all-weather coverage.

[0056] In addition, most of the intelligent intercom solutions in related technologies are based on WebRTC (Web Real-Time Communication) or proprietary protocols, which cannot be directly connected to post-national standard monitoring equipment in terms of signaling interaction or media stream format.

[0057] To address the aforementioned issues, this application proposes a two-way intercom link based on a national standard protocol (such as GB28181), aiming to bridge the aforementioned technological gap.

[0058] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0059] Figure 1 A schematic diagram of a scenario is shown, illustrating an intercom method or intercom device applicable to a kitchen environment that can be applied to embodiments of this disclosure.

[0060] Please refer to Figure 1 The diagram illustrates an implementation environment provided by an exemplary embodiment of this disclosure.

[0061] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0062] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0063] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0064] Server 105 can implement the following methods: receiving message sequences from the kitchen monitoring terminal; wherein the message sequence contains payload data using the MPEG-2 program stream encapsulation format; based on the encapsulation header identifier in the payload data, converting the payload data into a distribution data stream conforming to the streaming media protocol and distributing it to the client, which is used by the client to play video footage and audio of the kitchen environment; acquiring an intercom audio stream for the monitoring content in the payload data; wherein the encoding format of the intercom audio stream matches the audio encoding format in the payload data; and in response to the intercom command sent by the client, pushing the intercom audio stream to the kitchen monitoring terminal through the voice feedback link established with the kitchen monitoring terminal.

[0065] A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery) servers. It can also consist of multiple servers and can have any number of terminal devices, networks and servers depending on actual needs.

[0066] Under the above system architecture, this disclosure provides an intercom method for use in a kitchen environment, which can be executed by any electronic device with computing power.

[0067] Figure 2 This is a flowchart illustrating an intercom method applied in a kitchen environment according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power, for example, the method can be implemented by the above-described... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0068] In some embodiments, the intercom method described above for use in a kitchen environment can be performed by a media service.

[0069] Reference Figure 2 The intercom method for use in a kitchen environment provided in this embodiment may include the following steps.

[0070] Step S202: Receive the message sequence from the kitchen monitoring terminal; wherein the message sequence contains payload data in MPEG-2 program stream encapsulation format.

[0071] In some embodiments, the aforementioned kitchen monitoring terminal may be a kitchen national standard monitoring camera, or it may be a device that includes a kitchen national standard camera.

[0072] In some embodiments, the kitchen monitoring terminal can both collect monitoring videos of the kitchen environment and play the collected monitoring videos; this application does not impose any restrictions on this.

[0073] "National standard kitchen camera" refers to a surveillance camera deployed in the kitchen of a restaurant that conforms to national standards (such as GB / T 28181).

[0074] In some embodiments, the kitchen monitoring terminal can directly access the supervision platform and stably push real-time audio and video streams. However, due to the relatively closed protocol standards, it usually only supports one-way monitoring and cannot directly realize two-way intercom or intelligent interaction.

[0075] In some embodiments, the core difference between kitchen monitoring terminals and ordinary network cameras lies in their protocol standards, design goals, and application ecosystems. Kitchen monitoring terminals strictly adhere to national standards (such as GB / T 28181), with seamless integration with government regulatory platforms as their core objective. Their design emphasizes stability, compliance, and traceability, and they typically possess adaptability to kitchen environments such as oil stain and fog resistance. However, functionally, they natively only support one-way audio and video transmission, making it difficult to directly integrate internet-based intelligent interactive functions. Ordinary network cameras (such as home or commercial cameras) mostly use proprietary or industry-standard protocols, designed to meet flexible and diverse monitoring needs. They emphasize intelligent functions (such as motion detection and human recognition) and convenient internet access capabilities, but their protocol compatibility and regulatory compliance often cannot directly meet the integration requirements of national-level food safety regulatory platforms. Simply put, kitchen monitoring terminals are "born for regulation," while ordinary cameras are "born for general monitoring."

[0076] In some embodiments, standard audio and video streams conforming to national standards can be received from the kitchen monitoring terminal.

[0077] In some embodiments, the audio and video data streams collected by the kitchen monitoring terminal can be encapsulated and transmitted using the MPEG-2 Program Stream (PS) encapsulation format.

[0078] PS (Program Stream) is a media container format defined in the MPEG-2 standard, used to encapsulate audio and video data in relatively reliable transmission environments (such as optical disc storage and local files).

[0079] In some embodiments, the media service can provide services to one or more kitchen monitoring terminals.

[0080] In some embodiments, the media service can serve multiple cameras distributed in a distributed manner.

[0081] In some embodiments, a CDN node can serve one or more kitchen monitoring terminals in the vicinity.

[0082] In some embodiments, the media service can receive audio and video streams corresponding to the kitchen monitoring terminal from at least one CDN (Content Delivery Network) node.

[0083] In some embodiments, the media service may include a national standard media service and a live media service. The process of receiving the message sequence from the kitchen monitoring terminal may include: the national standard media service receiving national standard audio and video streams from the kitchen monitoring terminal from multiple uplink CDN nodes, wherein one uplink CDN node serves at least one kitchen monitoring terminal.

[0084] Step S204: Based on the encapsulation header identifier in the payload data, the payload data is converted into a distribution data stream conforming to the streaming media protocol and distributed to the client. The client is used to play video footage and audio of the kitchen environment.

[0085] In some embodiments, the starting position and type of video and audio stream packets in the MPEG-2 program stream encapsulation format can be identified and located based on the encapsulation header identifier in the payload data. Then, according to the payload data type indicated by the encapsulation header identifier, the video encoded stream and audio encoded stream are extracted from the message sequence respectively. Finally, the extracted video stream and / or audio stream are re-encapsulated according to the encapsulation specifications of streaming media protocols (such as RTMP, HTTP-FLV) to generate a distribution data stream that conforms to the streaming media transmission standard, and distributed to the client for playback of the video images and audio voice of the kitchen environment.

[0086] In some embodiments, audio and video streams (payload data) conforming to national standards can be converted into standard live streams conforming to live streaming specifications (distribution data streams conforming to streaming media protocols) and pushed to the live streaming system so that clients can pull and play the standard live stream, thereby playing real-time images and real-time audio of the kitchen to users.

[0087] In some embodiments, a "standard live stream conforming to live streaming specifications" can refer to an audio and video stream that uses common Internet transmission protocols such as RTMP (Real-Time Messaging Protocol) or HTTP-FLV (HTTP-based Flash Video) and is encapsulated in standard formats such as FLV (Flash Video).

[0088] In some embodiments, a "standard live stream conforming to live streaming specifications" can refer to a stream generated by converting a video stream output from a kitchen monitoring terminal that cannot be directly recognized by an internet player in real time through a protocol conversion service. Its core features are high compatibility and low latency, and it can be directly processed and played by mainstream CDN networks, live streaming systems, and client players.

[0089] In some embodiments, the kitchen monitoring terminal can push audio and video streams to media services based on the GB28181 national standard protocol and the RTP (Real-time Transport Protocol).

[0090] In some embodiments, because the GB / T compliant devices use the PS encapsulation format, which differs from industry-standard live streaming systems, a specific player is required to play this media stream, greatly limiting its use cases. Therefore, media services can perform protocol conversion to transform the PS-encapsulated GB / T stream into an RTMP standard live stream.

[0091] In some embodiments, the converted standard live stream can be pushed to a live streaming system that supports multiple downlink distribution formats (such as FLV, HLS, RTMP, etc.), is compatible with most players and clients, and meets the playback needs of different scenarios. In specific implementations, the client can choose to use the HTTP-FLV protocol to pull the live stream from the live streaming system for real-time video monitoring. HTTP-FLV is a streaming protocol with low latency, which makes the intercom process smoother.

[0092] In some embodiments, the client can selectively pull specified standard live streams from the live streaming system according to the user's needs.

[0093] The above steps, through a protocol conversion mechanism, convert the closed GB28181 stream of the kitchen monitoring terminal into a standard live stream that is universally used on the Internet in real time. This breaks down the technical barriers between professional national standard monitoring equipment and public live streaming platforms, enabling the Internet-based, low-latency, and cross-platform playback of monitoring images. This lays the foundation for large-scale, convenient public supervision and subsequent interaction.

[0094] Step S206: Obtain the intercom audio stream for the monitored content in the payload data; wherein the encoding format of the intercom audio stream matches the audio encoding format in the payload data.

[0095] In some embodiments, the aforementioned intercom audio stream can be acquired from real-time voice input by a real user, or it can be automatically synthesized by the system based on preset logic or an artificial intelligence model. This disclosure does not limit the specific method of generating the intercom audio stream.

[0096] In some embodiments, users can view the monitoring content collected by the kitchen monitoring terminal through the client, and then initiate a conversation with the kitchen monitoring terminal through the client regarding the monitoring content.

[0097] In some embodiments, a user-initiated voice chat stream for the monitored content can be acquired.

[0098] In some embodiments, acquiring an intercom audio stream for the monitored content in the payload data may include: in response to a voice intercom request initiated by a client, establishing an uplink long connection with the client using the WebSocket long connection protocol; and receiving an intercom audio stream for the monitored content in the payload data from the client through the uplink long connection.

[0099] In some embodiments, when the encoding format of the intercom voice stream does not match the audio encoding format in the payload data, the intercom voice stream can be converted to a format that matches the audio encoding format in the payload data (such as an audio format that conforms to national standards).

[0100] In step S208, in response to the intercom command sent by the client, an intercom audio stream is pushed to the kitchen monitoring terminal through the voice feedback link established with the kitchen monitoring terminal.

[0101] In some embodiments, in response to a voice intercom request initiated by the client (which may be an intercom request initiated by the user through the client), the kitchen monitoring terminal may be controlled to request the media service to establish a first downlink long connection (i.e., a voice backhaul link).

[0102] In some embodiments, after the first downlink long connection is established, the intercom voice stream can be sent to the kitchen monitoring terminal through the first downlink long connection so that the kitchen monitoring terminal can play the intercom voice stream to achieve two-way intercom.

[0103] In some embodiments, the format-converted intercom voice stream can be sent to the kitchen monitoring terminal via a first downlink long connection so that the kitchen monitoring terminal can play the intercom voice stream in real time for two-way intercom.

[0104] In the "transparent kitchen" monitoring aspect of related technologies, food delivery platforms periodically retrieve and analyze footage from the kitchen to identify abnormal behavior or unsanitary conditions, resulting in penalties and account suspensions. This process is reactive; by the time the abnormal behavior by kitchen staff occurs, it has already ended, making timely intervention impossible.

[0105] In some embodiments, the technical solution proposed in this application can be applied to the open kitchen scenario. Two connections can be established with the camera device in accordance with national standards. The uplink connection can be used to obtain the back kitchen video and audio from the camera in real time for identification and processing. In case of abnormal behavior, timely voice intervention can be carried out through the downlink connection.

[0106] The aforementioned technical solution achieves low-latency, highly reliable voice interaction between the client and the kitchen monitoring terminal by constructing a two-way intercom link that integrates protocol conversion and intelligent scheduling. Its core effects are twofold: firstly, it converts the closed audio and video streams output by the kitchen monitoring terminal into a live stream of universal internet standards in real time, breaking down protocol barriers and enabling internet-based, cross-platform live streaming of the monitoring footage; secondly, by establishing a manageable voice feedback link, it achieves real-time and accurate transmission of the client's voice stream to national standard equipment, completing the upgrade from one-way monitoring to two-way intelligent intercom capabilities, thus providing a complete technical foundation for real-time communication and intelligent operation in the "transparent kitchen" scenario.

[0107] In some embodiments, the above-mentioned voice backhaul link can be constructed by the following method: in response to the intercom command, a free port is dynamically allocated from the pre-configured port pool as the server listening port of the voice backhaul link; the address information of the server listening port is sent to the kitchen monitoring terminal so that the kitchen monitoring terminal can initiate a connection request to the server listening port to establish the voice backhaul link.

[0108] In some embodiments, a port pool may be maintained in advance, which may include multiple ports.

[0109] In some embodiments, when there is a need to communicate with the kitchen monitoring terminal (such as a user's intercom request initiated through the client or an intercom request automatically initiated by the background artificial intelligence based on the monitoring content), an idle port will be dynamically allocated from the pre-configured port pool to the kitchen monitoring terminal as the server listening port for the voice backhaul link of the kitchen monitoring terminal.

[0110] In some embodiments, one intercom session may correspond to one port, and one kitchen monitoring terminal may correspond to multiple ports.

[0111] In some embodiments, the port can be reclaimed when the intercom ends.

[0112] The above solution, by introducing a dynamic port pool mechanism, achieves on-demand allocation and efficient reuse of the voice backhaul link: when there is a need for intercom, an idle port is dynamically allocated from the pre-configured port pool as the server's listening port, and the port address is sent to the kitchen monitoring terminal to initiate a connection. This mechanism ensures that each intercom session has a dedicated communication port, avoiding data routing conflicts and flow chaos caused by multiple audio streams sharing the same port; at the same time, port resources are allocated on demand and recycled after use, supporting elastic expansion in high-concurrency scenarios, effectively improving the system's resource utilization and concurrent processing capabilities. In addition, one kitchen monitoring terminal can correspond to multiple ports, supporting multi-channel concurrent voice interaction needs, providing a stable and reliable audio transmission channel for multiple business scenarios in the kitchen environment (such as user intercom, violation alarms, order notifications, etc.).

[0113] In some embodiments, during the process of pushing the intercom audio stream to the kitchen monitoring terminal, the real-time audio stream in the monitoring content can also be detected in real time; in response to the detection of new voice activity from the real-time audio stream, an interruption event is triggered; in response to the interruption event, at least one of the following operations is performed: interrupting the push of the intercom audio stream to the kitchen monitoring terminal; clearing unsent audio frames in the audio buffer; and starting to acquire and push a new intercom audio stream corresponding to the new voice activity.

[0114] In some embodiments, after an interruption event is triggered, all generation and pushing actions for the current intercom voice stream will cease. For example, the speech synthesis module will no longer synthesize the intercom voice stream, or the pushing of the intercom audio stream to the kitchen monitoring terminal will be interrupted, etc. This application does not limit this.

[0115] The aforementioned method, by introducing an interruption event mechanism, detects user voice activity in real time during the process of pushing the intercom audio stream to the kitchen monitoring terminal. Once a new voice input is detected, an interruption event is immediately triggered, halting the current audio stream push, clearing unsent audio buffers, and quickly switching to the intercom audio stream corresponding to the new user's voice. This mechanism enables users to interrupt and interject in real time into the AI's voice output, effectively avoiding the interaction delays of traditional solutions where "users must wait for the AI ​​to finish speaking before continuing the conversation," significantly improving the natural fluency and user experience of the intercom process. Simultaneously, through a unified event (interruption event) driven architecture, it ensures coordinated switching between modules such as speech synthesis and audio push, avoiding playback chaos or resource waste caused by inconsistent states across multiple modules.

[0116] In some embodiments, pushing an intercom audio stream to a kitchen monitoring terminal may include the following methods: obtaining the current system time and the starting push system time for pushing the intercom audio stream to the kitchen monitoring terminal; determining the difference between the current system time and the starting push system time; determining the audio index of the audio frame that should be sent in the intercom audio stream based on the starting push system time and the difference, as the first audio frame index; extracting the audio frame corresponding to the first audio frame index from the audio buffer and pushing it to the kitchen monitoring terminal.

[0117] In some embodiments, system time may refer to the current time value maintained and provided by the operating system of the electronic device performing the method of this application (such as the server where the media service is located).

[0118] It is understood that those skilled in the art may also choose other time bases as the system time according to the actual application scenario, and this application does not impose strict restrictions on this.

[0119] This embodiment introduces an absolute timeline push mechanism based on system time. When pushing intercom audio streams to the kitchen monitoring terminal, the system time at the start of the push is recorded as a reference, and the current system time is acquired in real time during the push process. The index of the audio frame to be sent is determined by calculating the time difference, and the corresponding audio frame is extracted from the buffer for push. This mechanism ensures that the audio push speed is strictly synchronized with the physical time elapsed. Regardless of fluctuations in the speech synthesis generation speed or network transmission status, it avoids audio playback acceleration, stuttering, or frame loss caused by processing delays, ensuring smooth and accurate audio playback on the kitchen monitoring terminal. Furthermore, this mechanism does not rely on a specific time reference implementation method, offering good deployment flexibility and adaptability.

[0120] In some embodiments, pushing an intercom audio stream to a kitchen monitoring terminal includes: obtaining a first video timestamp corresponding to the monitoring content played by the client when the intercom is triggered; obtaining a second video timestamp corresponding to the monitoring content currently being played by the client; calculating the time offset between the second video timestamp and the first video timestamp; determining a target audio frame corresponding to the time offset from the intercom audio stream based on the time offset, wherein the start playback time of the intercom audio stream corresponds to the first video timestamp; extracting the target audio frame from the audio buffer, associating the target audio frame with the second video timestamp, and pushing it to the kitchen monitoring terminal so that the kitchen monitoring terminal synchronously plays the target audio frame when playing the monitoring video screen corresponding to the second video timestamp.

[0121] A video timestamp can refer to the playback progress of a frame in a video. This timestamp can be independent of a clock and can represent "at what second a certain frame should appear".

[0122] The currently playing monitoring content can refer to the real-time video and audio of the kitchen environment that the client is currently presenting to the user.

[0123] In some embodiments, kitchen staff can view kitchen monitoring content in real time through a kitchen monitoring terminal. The monitoring footage played on the terminal may have a certain delay (e.g., 2-5 seconds) compared to the client.

[0124] In a specific application scenario, when a supervisor is watching the monitoring content through a client, they discover a violation in the kitchen at the video timestamp of 12:00:00 and trigger the intercom. The system then initiates streaming processing: for each segment of speech uttered by the supervisor, the system identifies, understands, and synthesizes the corresponding intercom audio segment in real time, immediately associating the segment with the current video timestamp (e.g., 12:00:00.2) and pushing it to the kitchen monitoring terminal. Regardless of the playback delay on the kitchen terminal, when it reaches the video timestamp of 12:00:00.2, it will automatically play the corresponding audio segment (e.g., "Attention"); when it reaches 12:00:00.4, it will play "Notice," and so on. Thus, the progress of the supervisor's speech, the progress of the video content, and the progress of the audio heard in the kitchen are completely synchronized, achieving a precise audio-visual synchronized intercom effect where "what is said at the moment it is heard in the moment it is heard in the moment it is seen in the video."

[0125] This solution achieves audio-visual synchronization across edge-side playback latency by precisely binding the intercom audio stream to the inherent playback progress of the kitchen monitoring video. Regardless of the latency between the kitchen monitoring terminal and the client, the audio always aligns with the playback timestamp of the video content itself—the kitchen staff hear the corresponding intercom content in the same frame when the user issues a voice command. Simultaneously, this video timestamp-based association push mechanism naturally ensures that the playback order of audio frames strictly matches the video progress. Even during network fluctuations or streaming processing, there will be no issues with audio frame disorder, misalignment, or playback chaos, ensuring that the voice heard by kitchen staff is smooth, coherent, and accurately timed, significantly improving the stability and interactive experience of remote intercom in the kitchen.

[0126] In some embodiments, the aforementioned audio and video transmission links can employ dynamic CDN acceleration and proximity access strategies. The live streaming system can dynamically select the optimal CDN node for stream forwarding based on the user's geographical location and network conditions, while also supporting adaptive bitrate adjustment and automatic reconnection mechanisms to ensure the continuity and reliability of the stream.

[0127] In some embodiments, the live streaming system may include multiple CDN nodes. The process of converting the national standard audio and video stream into a standard live stream conforming to live streaming specifications and pushing it to the live streaming system so that clients can pull and play the standard live stream, thereby displaying real-time kitchen footage and audio to users, includes: distributing the standard live stream to multiple CDN nodes so that clients can dynamically pull and play the standard live stream from multiple CDN nodes based on a proximity access policy.

[0128] Converting the national standard audio and video stream into a standard live stream that conforms to live streaming specifications and pushing it to the live streaming system may include: the national standard media service converting the national standard audio and video stream into a standard live stream that conforms to live streaming specifications; the national standard media service sending the standard live stream to the live streaming media service; and the live streaming media service pushing the standard live stream to at least one downlink CDN node in the live streaming system, so that the client can pull and play the standard live stream from at least one downlink CDN node based on the nearest access policy.

[0129] Figure 3 This is a flowchart illustrating an intercom method applied in a kitchen environment according to an exemplary embodiment.

[0130] refer to Figure 3 The above-mentioned intercom method applied to the kitchen environment may include the following steps.

[0131] Step S302: Receive standard audio and video streams from kitchen monitoring terminals from multiple uplink CDN nodes, where one uplink CDN node serves at least one kitchen monitoring terminal.

[0132] In some embodiments, the aforementioned national standard audio and video stream may be a message sequence sent by a kitchen monitoring terminal, which may contain payload data using the MPEG-2 program stream encapsulation format.

[0133] Figure 4 This is a schematic diagram of a live streaming link architecture according to an exemplary embodiment.

[0134] like Figure 4 As shown, the above live streaming link can include multiple CDN nodes, which can include uplink CDN nodes and downlink CDN nodes. It can be understood that a CDN node can act as an uplink CDN node to receive standard video streams from the kitchen monitoring terminal, and also as a downlink CDN node to send live video streams to the client.

[0135] like Figure 4 As shown, the GB / T media access layer in the media service can receive GB / T audio and video streams from kitchen monitoring terminals from multiple uplink CDN nodes, with one uplink CDN node serving at least one kitchen monitoring terminal.

[0136] Step S304: Convert the national standard audio and video stream into a standard live stream that conforms to the live streaming specifications and push the standard live stream to at least one downlink CDN node in the live streaming system so that the client can pull and play the standard live stream from at least one downlink CDN node based on the nearest access strategy, thereby playing real-time kitchen footage and real-time audio to the user.

[0137] like Figure 4 As shown, the live media service in the media service can convert national standard audio and video streams into standard live streams that conform to live streaming specifications and push the standard live streams to at least one downlink CDN node in the live streaming system. This allows clients to pull and play the standard live streams from at least one downlink CDN node based on a proximity access strategy, thereby providing users with real-time footage and audio from the kitchen.

[0138] Step S306: In response to the voice intercom request initiated by the client, control the kitchen monitoring terminal to request the media service to establish the first downlink long connection.

[0139] Step S308: After the first downlink long connection is established, receive the intercom voice stream from the client.

[0140] Step S310: Convert the intercom voice stream into an audio format that conforms to national standards.

[0141] Step S312: The converted intercom voice stream is sent to the kitchen monitoring terminal through the first downlink long connection so that the kitchen monitoring terminal can play the intercom voice stream in real time for two-way intercom.

[0142] This embodiment achieves efficient aggregation and intelligent distribution of massive amounts of kitchen monitoring terminal streams by introducing a layered CDN architecture and a long-connection mechanism. The specific technical effects are reflected in at least three aspects: First, by connecting uplink CDN nodes to cameras nearby and downlink CDN nodes dynamically serving clients, a highly scalable and low-latency "camera → cloud → user" live streaming path is constructed; second, by establishing a controlled first downlink long connection, the intercom voice stream is ensured to be accurately and in real-time transmitted to the target camera; third, while achieving two-way intercom functionality, the entire solution maintains full compatibility with national standard protocols, ultimately achieving comprehensive optimization of bandwidth, latency, and reliability for live monitoring and voice interaction in large-scale deployment scenarios.

[0143] Figure 5 This is a diagram illustrating a voice link structure according to an exemplary embodiment.

[0144] In some embodiments, the voice stream from the client to the kitchen monitoring terminal can be divided into an uplink and a downlink.

[0145] In some embodiments, receiving an intercom voice stream from a client may include receiving the intercom voice stream from the client via an uplink long connection on the uplink.

[0146] In some embodiments, the media service may respond to a voice call request initiated by the client by establishing an uplink long connection with the client using the WebSocket long connection protocol. For example... Figure 5 As shown, the uplink can receive the intercom voice stream from the sender through the WebSocket long connection protocol and upload it to its corresponding media service (such as media service A).

[0147] In some embodiments, the audio stream from the client to the kitchen monitoring terminal may be an audio stream in G711A encoding format.

[0148] In some embodiments, the uplink can employ the WebSocket long-connection protocol, where the client pushes sampled audio data to the intercom media service (such as media service A) in real time. Compared to the traditional HTTP polling method, WebSocket provides persistent bidirectional communication capabilities, with lower latency and higher transmission efficiency, meeting the stringent requirements of real-time intercom. At the implementation level, each intercom session can be assigned a unique session ID and stream ID. After authentication (such as security authentication), the client establishes a WebSocket connection and continuously pushes audio data according to a predetermined audio frame format (which can be 20ms per frame). Upon receiving the user's audio, the server performs cache management and stream forwarding.

[0149] In some embodiments, the aforementioned downlink can transmit the user's audio stream to the kitchen monitoring terminal device, for example, it can be implemented according to the GB28181 national standard. Figure 5 As shown, media service A can directly forward the user's video stream or indirectly forward it to the kitchen monitoring terminal device through media service B.

[0150] In some embodiments, a dynamic port listening mechanism can be used, with each audio stream (session) having its own dedicated TCP (Transmission Control Protocol) port. The core advantage of this design is that after a camera device is connected to the corresponding port, the media service can accurately identify the audio stream corresponding to that device and the corresponding kitchen monitoring terminal, avoiding the stream chaos problem caused by multiple devices sharing a single port.

[0151] In some embodiments, a port pool can be maintained to dynamically allocate and reclaim port resources, ensuring efficient port utilization in high-concurrency scenarios.

[0152] In some embodiments, intelligent resource reclamation can be performed, automatically releasing occupied port and memory resources when the intercom session ends to prevent resource leakage.

[0153] In some embodiments, the intercom method described above for use in kitchen environments can also support NAT (Network Address Translation) traversal and intranet traversal technologies, enabling it to adapt to complex network environments. For public network devices, the system directly exposes ports through public IP (Internet Protocol) hosts; for intranet devices, the system supports forwarding via an intranet traversal proxy. This design allows the system to support camera devices from different network environments, significantly reducing deployment complexity.

[0154] In some embodiments, the payload data can be audio retrieved and audio format converted to obtain target audio; speech recognition can be performed on the target audio to obtain recognized text information; the recognized text information can be input into a large language model so that the large language model can generate response text based on the recognized text information; a first prompt audio stream can be synthesized based on the response text; and the first prompt audio stream can be pushed to the kitchen monitoring terminal through the voice feedback link.

[0155] In some embodiments, kitchen business features can be extracted from the monitored content; the kitchen business features can be matched with vector data in a preset kitchen knowledge base to generate an audio stream that conforms to the kitchen business scenario based on the matched preset kitchen knowledge constraint big language model.

[0156] In some embodiments, the method may further include generating a second intercom audio stream by at least one of the following methods and pushing the second intercom audio stream to the kitchen monitoring terminal via a voice backhaul link.

[0157] Method 1: Detect kitchen violations in the monitored content; if a preset violation feature is detected, determine the preset alarm text corresponding to the violation feature; convert the preset alarm text into a second intercom audio stream and push it to the kitchen monitoring terminal.

[0158] Method 2: Obtain structured kitchen order data from the kitchen business system; based on preset trigger conditions, parse the structured kitchen order data to generate order prompt text instructions; input the order prompt text instructions into the speech synthesis model to generate a second intercom audio stream.

[0159] In some embodiments, user inquiries regarding orders can also be collected (e.g., when will my food be ready; please don't add chili to my order), and these inquiries can be parsed to generate inquiry text, which is then synthesized into a second intercom audio stream and sent to the kitchen for communication with the merchant. Specifically, after collecting user inquiries, the intelligent assistant (such as an intelligent robot) can intelligently handle the communication based on the user's inquiry domain, including product inquiries, centralized order reminders and notes, and quarterly after-sales follow-ups.

[0160] Figure 6 This is a flowchart illustrating an intelligent two-way dialogue method according to an exemplary embodiment.

[0161] refer to Figure 6 The above-mentioned intelligent two-way dialogue method may include the following steps.

[0162] Step S602: Extract audio from the GB standard audio and video stream and convert the audio format to obtain the target audio.

[0163] In some embodiments, audio components can be separated and processed from GB standard audio and video streams to achieve format adaptation.

[0164] Specifically, the audio basic stream can be extracted from the input GB standard audio and video stream first. This audio can be encapsulated in G.711A encoding format. Then, through decoding and resampling, it is converted into a common format (such as 16kHz, 16bit PCM format) required by subsequent speech recognition and synthesis modules, thereby generating standardized target audio and providing a unified and compatible input signal for intelligent speech processing.

[0165] In some embodiments, an FLV live stream can be pulled from the live streaming link in real time, and audio can be extracted and converted to output PCM (Pulse-Code Modulation) format audio.

[0166] Step S604: Perform speech recognition on the target audio to obtain the recognized text information.

[0167] In some embodiments, the converted PCM audio stream can be streamed into the ASR (Automatic Speech Recognition) module for real-time speech recognition. The ASR module can integrate VAD (Voice Activity Detection) functionality, automatically identifying the user's speaking boundaries. When a user begins to speak, the system creates a new dialogue session; when a user stops speaking, the system triggers the final ASR result output. Furthermore, the system supports confidence threshold configuration; only results with a confidence level exceeding the threshold (e.g., 0.85) are accepted, effectively improving recognition accuracy. The recognized text output by the ASR module includes complete timestamp information, providing a foundation for subsequent link tracing and performance analysis.

[0168] Figure 7 This is a schematic diagram of an intelligent media processing engine according to an exemplary embodiment.

[0169] like Figure 7 As shown, the kitchen monitoring terminal can monitor the kitchen in real time and acquire national standard audio and video (including message sequences containing payload data in MPEG-2 program stream encapsulation format). Then, the intelligent intercom media processing engine can perform FLV streaming, audio transcoding, and streaming ASR on the national standard audio and video to extract text information from it.

[0170] Step S606: Input the recognized text information into the large language model so that the large language model can generate response text based on the recognized user intent.

[0171] like Figure 7 As shown, text information obtained from the national standard audio and video streams in the kitchen can be input into a large language model (such as...). Figure 7 (In the context of intelligent dialogue agents), so that a large language model can provide a text response to the text information.

[0172] In some embodiments, the identified text can be fed into a Large Language Model (LLM) for inference, generating corresponding response text based on the identified user intent.

[0173] In some embodiments, to improve the accuracy and professionalism of responses, a domain knowledge base can be built for the large language model.

[0174] In some embodiments, the aforementioned domain knowledge base may cover professional knowledge related to open kitchen scenarios, such as ingredient information, dish preparation processes, and food safety standards.

[0175] In some embodiments, when the large model is reasoning, it can retrieve relevant information from the knowledge base and combine it with the user's question (identified text information) to generate a more accurate response.

[0176] In some embodiments, Function Calling capabilities can also be integrated, enabling large models to call external tools and APIs (Application Programming Interfaces), allowing them to not only answer static knowledge questions but also handle dynamic business queries.

[0177] In some embodiments, streaming output can also be supported, and the text generated by the large model can be returned word by word or sentence by sentence, further reducing the first word delay.

[0178] Step S608: Synthesize the response voice supported by the kitchen monitoring terminal based on the response text.

[0179] like Figure 7 As shown, the generated response text can be fed into a streaming TTS (Text-To-Speech) module for real-time speech synthesis.

[0180] In some embodiments, the TTS module can support multiple speakers and speech rate adjustments, allowing users to select appropriate timbres based on scenario requirements, such as a friendly female voice or a professional male voice. The system also supports emotional speech synthesis, which can adjust the emotional expression of the speech according to the text content, making AI responses more natural and fluent.

[0181] In some embodiments, the TTS module can employ streaming synthesis technology, starting synthesis immediately after text input without waiting for the complete text. The synthesized audio is in PCM format (e.g., 16bit, 16kHz) and output frame-by-frame as audio frames. Each audio frame carries complete tracking information (session ID, message ID, timestamp) to ensure accurate downstream processing and forwarding.

[0182] Step S610: Send the reply voice message to the kitchen monitoring terminal for playback.

[0183] In some embodiments, the synthesized speech (response speech) described above needs to be converted into a format that conforms to national standards. For example... Figure 7 As shown, synthesized speech in PCM format can be converted into speech in G711A format for playback on the kitchen monitoring terminal.

[0184] In some embodiments, a frame synchronization mechanism based on an absolute timeline can be employed to ensure the continuity and accuracy of the audio. Specifically, the system records the start time of the audio streaming, calculates the number of audio frames to be sent based on the current time, retrieves the corresponding data from the buffer, performs format conversion, and pushes the data. This design avoids audio stuttering or accelerated playback issues caused by network fluctuations or processing delays.

[0185] In some embodiments, when the audio data in the buffer is insufficient, the system (such as two-way intercom) will automatically supplement with silence data to ensure the continuity of the audio stream. After conversion, the audio stream can be pushed to the kitchen monitoring terminal device, where the camera decodes and plays it so that the kitchen staff can hear the AI's response. The entire streaming process supports an automatic reconnection mechanism; when the network is interrupted, the system will attempt to re-establish the connection and resume streaming.

[0186] This embodiment achieves real-time intelligent interactive upgrades to the national standard monitoring stream by constructing an end-to-end full-stream processing chain of "audio extraction → speech recognition → intelligent reasoning → speech synthesis". Its technical effects are specifically reflected in: converting national standard audio collected in the kitchen scene and packaged in G.711A format into processable PCM audio in real time and performing high-accuracy ASR recognition; relying on the capabilities of the domain knowledge base and large language model, performing contextual understanding and intelligent response generation on the recognized text; then synthesizing the text response into human-like speech through streaming TTS technology, and finally converting it back to G.711A format via the national standard protocol for playback on the camera. Thus, without any modification to existing national standard hardware, it achieves natural, fluent, and low-latency intelligent two-way voice dialogue.

[0187] Figure 8 This is a flowchart illustrating a voice playback method according to an exemplary embodiment.

[0188] In some embodiments, the response voice can be generated word by word and cached.

[0189] refer to Figure 8 The above-mentioned voice playback method may include the following steps.

[0190] Step S802: Determine the start time for pushing the stream to the kitchen monitoring terminal.

[0191] Step S804: Calculate the index of the audio frame to be sent based on the current time and the start time.

[0192] Step S806: Based on the audio frame index, extract the audio frame to be sent from the cache and convert it into an audio format supported by the kitchen monitoring terminal.

[0193] Step S808: Send the converted audio frame to the kitchen monitoring terminal for playback.

[0194] In some embodiments, retrieving the audio frame to be sent from the cache and converting it to an audio format supported by the kitchen monitoring terminal based on the audio frame index may include: if there is no audio frame corresponding to the audio frame index in the cache, generating silent data of the corresponding duration; and sending the silent data to the kitchen monitoring terminal for playback after format conversion, so as to maintain the continuous playback of the audio stream pushed to the kitchen monitoring terminal.

[0195] In some embodiments, the silence data can be artificially generated standard format audio frames with zero amplitude, used to fill the audio stream during silent periods or when data processing is delayed. Its core function is to maintain the continuity, synchronization and format integrity of the real-time audio stream, and to prevent playback stuttering, clock out of sync or connection abnormalities caused by data interruption, thereby ensuring that (such as in AI dialogue scenarios) speech can be played smoothly, naturally and without interruption.

[0196] This embodiment solves the problem of playback discontinuity caused by fluctuations in streaming TTS synthesis rate and network transmission by using a frame synchronization and silence filling mechanism based on an absolute time axis. Its technical advantages are: by recording the start time of the streaming system and dynamically calculating the index of the audio frame to be sent, precise control of the audio playback progress is achieved; when the synthesis buffer fails to provide the corresponding audio frame in time, the system automatically inserts format-adapted silence data, ensuring the strict continuity and stability of the audio stream sent to the kitchen monitoring terminal in the time dimension. This effectively eliminates voice stuttering, acceleration, or interruption caused by processing delays or network jitter, ensuring the auditory fluency and natural experience of intelligent dialogue responses.

[0197] Figure 9 This is a flowchart illustrating a dialogue method according to an exemplary embodiment.

[0198] refer to Figure 9 The above-mentioned dialogue method may include the following steps.

[0199] Step S902: Perform real-time analysis of the national standard audio and video streams to detect violations in the kitchen environment.

[0200] In step S904, in response to the detection of a violation, the control system actively establishes a second long connection between the kitchen monitoring terminal and the media service.

[0201] Step S906: Synthesize a violation warning voice that conforms to national standards.

[0202] Step S908: The violation warning voice message is sent to the kitchen monitoring terminal via the second long connection for playback, so as to provide a voice alarm on-site in the kitchen.

[0203] For example, in routine monitoring of a restaurant kitchen, real-time analysis of the video stream from the kitchen monitoring terminal can identify a staff member not wearing a chef's hat and handling ready-to-eat food directly with their hands, classifying this as a food safety violation. In response, the kitchen monitoring terminal can be immediately and proactively notified, establishing a dedicated second-line long-lived connection with the cloud media service. Subsequently, a clear alarm voice conforming to national standard audio formats is automatically synthesized: "Warning: Unprofessional attire and hand contact with ready-to-eat food detected. Please correct immediately!" This is then transmitted in real-time to the kitchen monitoring terminal for playback. This allows the violator to hear a clear voice warning immediately, prompting them to correct their behavior. This upgrades the regulatory model from passive video recording to proactive real-time intervention, effectively improving the standardization of kitchen operations and food safety levels.

[0204] This embodiment combines intelligent video analysis with a proactive voice alarm mechanism to achieve real-time intervention and on-site alerts for violations in the kitchen. Its technical effects include at least: automatically identifying violations in the video stream and proactively triggering a dedicated long-term connection with the kitchen monitoring terminal to simultaneously generate and send out warning voice messages conforming to national audio standards. This upgrades traditional passive and delayed video surveillance to immediate and proactive on-site voice alarms, significantly improving the timeliness, intervention strength, and deterrent effect of kitchen safety supervision.

[0205] Figure 10 This is a flowchart illustrating a dialogue method according to an exemplary embodiment.

[0206] refer to Figure 10 The above-mentioned dialogue method may include the following steps.

[0207] Step S1002: Analyze the kitchen order information and determine whether voice communication with the kitchen is required based on the kitchen order analysis results.

[0208] Step S1004: Drive the kitchen monitoring terminal to initiate a third downlink long connection to the media service.

[0209] Step S1006: Based on the kitchen order information, synthesize an order prompt voice that is adapted to the kitchen monitoring terminal.

[0210] In step S1008, the order prompt voice is sent to the kitchen monitoring terminal for playback via the third downlink long connection to communicate order information with kitchen staff.

[0211] For example, during the lunchtime peak hours at a large takeout restaurant, the order analysis module identified that order A (with the note "no cilantro") had been backlogged for nearly 10 minutes without being prepared, while order B (an urgent order) was about to expire. If it was determined that the kitchen needed immediate attention, the system automatically initiated a dedicated third-party long connection from the kitchen monitoring terminal in that area to the media service. Then, based on these two urgent order information, a clear voice prompt was synthesized: "Please note: Order number 12, please omit cilantro; urgent order number 58, please process it first." This prompt was then transmitted in real-time to the corresponding kitchen monitoring terminal for amplification via the established long connection. This allowed busy chefs to hear the clear order instructions directly without having to look at the screen, enabling them to adjust their work order promptly and effectively preventing preparation errors and order delays.

[0212] This embodiment combines intelligent order analysis with proactive voice broadcasting to achieve efficient scheduling and precise intervention in the kitchen production process. Its technical effects include at least the ability to automatically identify anomalies in the order queue (such as backlogged orders, special notes, and rush orders) and proactively trigger a dedicated long-term connection channel with the kitchen monitoring terminal. This transforms key order information into clear, targeted voice commands in real time and broadcasts them to the kitchen. This effectively solves the pain points of information overload and inconvenient screen viewing during peak hours, upgrading the information transmission method from "manual viewing" to "voice delivery," thereby significantly improving the accuracy, timeliness, and overall operational efficiency of order processing.

[0213] The method for two-way communication with a kitchen monitoring terminal proposed in this application will be explained and described below with reference to specific embodiments.

[0214] In some embodiments, the method described above for two-way communication with the kitchen monitoring terminal can be divided into two steps: 1. Establishing a two-way communication link, which is for the correct and low-latency transmission of the communication voice stream; 2. Implementing intelligent dialogue on the two-way communication link. Specifically, it may include the following steps.

[0215] (a) Establishment of two-way intercom link.

[0216] Step 1: Audio and video transmission link from kitchen equipment to the audience (e.g., Figure 4 The live stream link is shown.

[0217] The kitchen monitoring terminal uses the GB28181 national standard protocol and pushes audio and video streams to the media service via the RTP protocol. Because the national standard equipment uses the PS encapsulation format, which differs from industry-standard live streaming systems, a specific player is required to play this media stream, greatly limiting its use cases. Therefore, the media service needs to perform protocol conversion, transforming the PS-encapsulated national standard stream into an RTMP standard live stream.

[0218] After the converted media stream is pushed to the live streaming system, the system supports multiple downlink distribution formats (FLV, HLS, RTMP, etc.), which are compatible with most players and clients on the market, meeting the playback needs of different scenarios. In terms of implementation, the client can choose to use the HTTP-FLV protocol to pull the live stream from the live streaming system for real-time video monitoring. HTTP-FLV is a streaming protocol with low latency, which makes the intercom process smoother.

[0219] To further reduce latency, the entire link can employ dynamic CDN acceleration and a proximity-based access strategy. The system can dynamically select the optimal CDN node for stream forwarding based on the user's geographical location and network conditions. It also supports adaptive bitrate adjustment and automatic reconnection mechanisms to ensure stream continuity and reliability. For details, please refer to [link / reference]. Figure 4 The illustrated embodiment.

[0220] Step 2: Audio transmission link from front-end devices to back-of-house devices (e.g., Figure 5 (The intercom link shown).

[0221] Compared to step 1, only the audio stream needs to be transmitted here, not the video. Specifically, the audio stream is transmitted in G711A encoding format. The audio stream transmission link can be divided into uplink and downlink.

[0222] like Figure 5 As shown, the uplink can use the WebSocket long-connection protocol, where the client pushes sampled audio data to the intercom media service in real time. Compared to the traditional HTTP polling method, WebSocket provides persistent bidirectional communication capabilities, with lower latency and higher transmission efficiency, meeting the stringent requirements of real-time intercom. At the implementation level, the system assigns a unique session ID and stream ID to each intercom session. After authentication, the client establishes a WebSocket connection and continuously pushes audio data according to a predetermined audio frame format (usually 20ms per frame). Upon receiving the user's audio, the server performs cache management and stream forwarding.

[0223] like Figure 5As shown, the downlink transmits the user's audio stream to the camera device, implemented according to the GB28181 national standard. The system employs a dynamic port listening mechanism, with each audio stream occupying a dedicated TCP port. The core advantage of this design is that after the camera device connects to the corresponding port, the media service can accurately identify the audio stream corresponding to that device, avoiding the stream chaos caused by multiple devices sharing a single port. Simultaneously, the system maintains a port pool, dynamically allocating and reclaiming port resources to ensure efficient port utilization in high-concurrency scenarios. The system also implements an intelligent resource reclamation mechanism, automatically releasing occupied port and memory resources when the intercom session ends, preventing resource leaks.

[0224] In some embodiments, the two-way intercom system can support NAT traversal and intranet traversal technologies, enabling it to adapt to complex network environments. For public network devices, the system directly exposes ports via public IP hosts; for intranet devices, the system supports forwarding via an intranet traversal proxy. This design allows the system to support camera devices from different network environments, significantly reducing deployment complexity.

[0225] (II) Implementation of the intelligent intercom system.

[0226] Once the bidirectional link is established, the transmission base is complete. The next step is to combine LLM (Large Language Model), ASR (Automatic Speech Responsiveness), TTS (Text-to-Speech), and other AI multimodal capabilities to create an intelligent conversational robot and integrate it into the intercom link for end-to-end intelligent dialogue. Figure 7 As shown, an intelligent dialogue system needs to implement the following four steps.

[0227] Step 1: Audio extraction and recognition.

[0228] The system pulls the FLV live stream from the live streaming link in real time, extracts and converts the audio, and outputs PCM format audio. This step is necessary because the audio stream is encoded in G711A format in GB28181 devices, while multimodal systems for speech recognition and speech synthesis generally support PCM format.

[0229] The converted PCM audio stream is fed into the ASR (Audio Recognition) module for real-time speech recognition. The ASR module integrates VAD (Voice Activity Detection) functionality, automatically recognizing the user's speaking boundaries. When a user begins to speak, the system creates a new dialogue session; when a user stops speaking, the system triggers the final ASR result output. Furthermore, the system supports confidence threshold configuration; only results with a confidence level exceeding the threshold (e.g., 0.85) are accepted, effectively improving recognition accuracy. The recognized text output by the ASR module includes complete timestamp information, providing a foundation for subsequent link tracing and performance analysis.

[0230] Step 2: LLM large model reasoning and knowledge enhancement.

[0231] The text identified by ASR (Automatic Speech Recognition) is fed into a Large Language Model (LLM) for inference, generating corresponding response text based on the identified user intent. To improve the accuracy and professionalism of the responses, the system has built a domain knowledge base covering professional knowledge related to the transparent kitchen scenario, such as ingredient information, food preparation processes, and food safety regulations. During inference, the LLM retrieves relevant information from the knowledge base and combines it with the user's question to generate a more accurate response.

[0232] The system also integrates Function Calling capabilities, enabling large models to call external tools and APIs, allowing them to not only answer static knowledge questions but also handle dynamic business queries.

[0233] The processing time for large model inference depends on the model's complexity, knowledge base retrieval efficiency, and network latency, typically ranging from 500 to 1000 ms in actual tests. The system supports streaming output, allowing text generated by large models to be returned character by character or sentence by sentence, further reducing first-character latency.

[0234] Step 3: Speech synthesis and voice customization.

[0235] The generated response text is fed into a streaming TTS (text-to-speech) module for real-time speech synthesis. The TTS module supports multiple speakers and adjustable speech rate, allowing users to select appropriate timbres based on the scenario, such as a friendly female voice or a professional male voice. The system also supports emotional speech synthesis, which adjusts the emotional expression of the speech according to the text content, making the AI ​​response more natural and fluent.

[0236] The TTS module employs streaming synthesis technology, starting synthesis immediately after text input without waiting for the complete text. The synthesized audio is in PCM format (16bit, 16kHz) and output frame-by-frame. Each audio frame carries complete tracking information (session ID, message ID, timestamp) to ensure accurate downstream processing and forwarding.

[0237] Step 4: Format conversion and intercom streaming.

[0238] Synthesized speech in PCM format needs to be converted to G.711A format to comply with the GB28181 national standard. The system employs a frame synchronization mechanism based on an absolute time axis to ensure audio continuity and accuracy. Specifically, the system records the start time of the audio stream, calculates the number of audio frames to be sent based on the current time, retrieves the corresponding data from the buffer, performs format conversion, and pushes the data. This design avoids audio stuttering or accelerated playback issues caused by network fluctuations or processing delays.

[0239] When the audio data in the buffer is insufficient, the system automatically supplements with silence data to ensure the continuity of the audio stream. After conversion, the audio stream can be pushed to the camera device, which decodes and plays it, allowing the kitchen staff to hear the AI's response. The entire streaming process supports an automatic reconnection mechanism; if the network is interrupted, the system will attempt to re-establish a connection and resume streaming. For details, please refer to [link / reference]. Figure 7 The illustrated embodiment.

[0240] In some embodiments, the entire processing chain described above can adopt a full-streaming design, supporting streaming processing at each stage from FLV streaming, ASR identification, TTS synthesis to intercom streaming, avoiding buffering delays. The core advantage of the streaming design is that data can be immediately transmitted to downstream processing once it is generated, without waiting for complete data blocks, making the intercom process sufficiently smooth.

[0241] In some embodiments, the above method can be implemented based on a Pipeline architecture, where each processing stage (FLV streaming, audio conversion, ASR, TTS, and intercom streaming) is implemented as an independent plugin. Plugins communicate with each other using a Publisher / Subscriber pattern; the publisher plugin publishes its processing results to a designated Track, and the subscriber plugin subscribes to data from the Track for further processing. This design achieves complete decoupling between plugins, supporting flexible business customization and rapid iteration.

[0242] In some embodiments, the above method can also employ an event-driven asynchronous processing approach, with each plugin communicating and interacting via an event bus. The event bus supports various event types, including system events (such as plugin startup and shutdown), business events (such as ASR recognition completion and TTS synthesis completion), and control events (such as user interruption and session timeout). When ASR recognition is complete, an EventDataChatBase event is published, which the business backend listens for and triggers large-scale model inference. When TTS synthesis is complete, another event is published, which the intercom streaming plugin listens for and performs audio conversion and streaming. This event-driven design improves system throughput and response speed, supporting stable operation in high-concurrency scenarios.

[0243] In some embodiments, the above method can also support user interruption and VAD (Voice Activity Detection) timeout management, providing a natural and smooth dialogue experience. When the user speaks again while the AI ​​is speaking, the VAD module detects new voice activity and immediately issues an interruption event. After the TTS plugin and the intercom streaming plugin detect the interruption event, they cancel the current synthesis and streaming tasks, clear the audio queue, and immediately begin processing new user input. This interruption mechanism simulates natural interruption behavior in human conversation, avoiding the awkward experience of users having to wait for the AI ​​to finish speaking before continuing the conversation.

[0244] In some embodiments, the method described above can also assign a unique tracking ID (TalkID) and message ID (MsgID) to each intercom session, recording timestamps for each stage of the process from the user speaking to the AI's response. Tracking information is transmitted between plugins along with audio frames and events. Each plugin records its current timestamp during processing and reports it to the monitoring system. This enables the system to perform multi-dimensional performance analysis, including ASR recognition latency, large model inference latency, TTS synthesis latency, and end-to-end total latency, facilitating problem localization and optimization. The system also supports distributed tracking, enabling the tracking of a complete conversation across multiple services and components.

[0245] The above method integrates the GB28181 national standard video surveillance protocol with the RTMP / HTTP-FLV streaming media transmission protocol, constructing a two-way dialogue channel between the client and the kitchen monitoring terminal. Based on this, by integrating AI multimodal technologies such as streaming ASR speech recognition, streaming TTS speech synthesis, large language model (LLM), and knowledge base, end-to-end intelligent dialogue capabilities are achieved. This solves the problem of traditional intelligent dialogue solutions based on proprietary protocols being incompatible with GB28181 national standard equipment, providing a complete intelligent intercom solution for transparent kitchen scenarios.

[0246] In some embodiments, the technical solutions proposed in this application can be applied to at least the following fields.

[0247] Food safety supervisor: When a violation is detected by kitchen staff, communicate with the staff in real time and urge them to correct the violation.

[0248] Order issue consultation: After collecting user issues, we communicate with merchants to resolve them, including product inquiries, order reminders during the sales process, and after-sales progress tracking.

[0249] Store Management: Artificial intelligence (AI) store managers can use this intercom capability to communicate with merchants about daily affairs.

[0250] It should be particularly noted that the steps in each embodiment of the intercom method applied to the kitchen environment described above can be interchanged, substituted, added, or deleted from each other. Therefore, these reasonable permutations and combinations of the intercom method applied to the kitchen environment should also fall within the protection scope of this disclosure, and the protection scope of this disclosure should not be limited to the embodiments.

[0251] It should be noted that the scope of protection of this application should include, but is not limited to, the specific implementation methods described in the embodiments. Any alternative solution that uses a different name but substantially performs the same function and achieves the same technical effect falls within the scope of protection defined by the claims of this application.

[0252] Based on the same inventive concept, this disclosure also provides an intercom device for use in a kitchen environment, as shown in the following embodiment. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiment, the implementation of this device embodiment can refer to the implementation of the above-described method embodiment, and repeated details will not be elaborated further.

[0253] Figure 11 This is a block diagram illustrating an intercom device applied in a kitchen environment according to an exemplary embodiment. (Refer to...) Figure 11 The intercom device 1100 for use in a kitchen environment provided in this embodiment may include the following modules.

[0254] The message receiving module 1101 is used to receive a message sequence from the kitchen monitoring terminal; wherein the message sequence contains payload data using the MPEG-2 program stream encapsulation format; the conversion module 1102 is used to convert the payload data into a distribution data stream conforming to the streaming media protocol based on the encapsulation header identifier in the payload data and distribute it to the client, wherein the client is used to play the video footage and audio of the kitchen environment; the voice stream acquisition module 1103 is used to acquire an intercom audio stream for the monitoring content in the payload data; wherein the encoding format of the intercom audio stream matches the audio encoding format in the payload data; the voice feedback module 1104 is used to respond to the intercom command sent by the client and push the intercom audio stream to the kitchen monitoring terminal through the voice feedback link established with the kitchen monitoring terminal.

[0255] In some embodiments, the intercom device 1100 applied in the kitchen environment may further include: a port allocation module and a port address distribution module.

[0256] The port allocation module can be used to dynamically allocate an idle port from the pre-configured port pool in response to the intercom command, as the server listening port of the voice backhaul link; the port address distribution module can be used to send the address information of the server listening port to the kitchen monitoring terminal, so that the kitchen monitoring terminal can initiate a connection request to the server listening port and establish the voice backhaul link.

[0257] The intercom device 1100 applied in the kitchen environment may also include: an implementation detection module, an interruption event triggering module, and an interruption event response module.

[0258] The detection module can be used to detect the real-time audio stream in the monitored content during the process of pushing the intercom audio stream to the kitchen monitoring terminal; the interruption event triggering module can be used to trigger an interruption event in response to the detection of new voice activity in the real-time audio stream; the interruption event response module can be used to respond to the interruption event and perform at least one of the following operations: interrupt the push of the intercom audio stream to the kitchen monitoring terminal; clear the unsent audio frames in the audio buffer; and start acquiring and pushing a new intercom audio stream corresponding to the new voice activity.

[0259] In some embodiments, pushing the intercom audio stream to the kitchen monitoring terminal may include: obtaining the current system time and the starting push system time for pushing the intercom audio stream to the kitchen monitoring terminal; determining the difference between the current system time and the starting push system time; determining the audio index of the audio frame that should be sent in the intercom audio stream based on the starting push system time and the difference, as the first audio frame index; extracting the audio frame corresponding to the first audio frame index from the audio buffer and pushing it to the kitchen monitoring terminal.

[0260] In some embodiments, pushing an intercom audio stream to a kitchen monitoring terminal may include: obtaining a first video timestamp corresponding to the monitoring content played by the client when the intercom is triggered; obtaining a second video timestamp corresponding to the monitoring content currently being played by the client; calculating the time offset between the second video timestamp and the first video timestamp; determining a target audio frame corresponding to the time offset from the intercom audio stream based on the time offset, wherein the start playback time of the intercom audio stream corresponds to the first video timestamp; extracting the target audio frame from the audio buffer, associating the target audio frame with the second video timestamp, and pushing it to the kitchen monitoring terminal so that the kitchen monitoring terminal synchronously plays the target audio frame when playing the monitoring video screen corresponding to the second video timestamp.

[0261] In some embodiments, acquiring an intercom audio stream for the monitored content in the payload data may include: in response to a voice intercom request initiated by the client, establishing an uplink long connection with the client using the WebSocket long connection protocol; and receiving an intercom audio stream for the monitored content in the payload data from the client through the uplink long connection.

[0262] In some embodiments, the intercom device 1100 applied in the kitchen environment may further include: a target audio acquisition module, a text information recognition module, a reply text generation module, a first prompt audio stream synthesis module, and a first prompt audio stream push module.

[0263] The target audio acquisition module can be used to extract audio from the payload data and convert the audio format to obtain target audio; the text information recognition module can be used to perform speech recognition on the target audio to obtain recognized text information; the response text generation module can be used to input the recognized text information into a large language model so that the large language model can generate response text based on the recognized text information; the first prompt audio stream synthesis module can be used to synthesize a first prompt audio stream based on the response text; and the first prompt audio stream push module can be used to push the first prompt audio stream to the kitchen monitoring terminal through the voice feedback link.

[0264] In some embodiments, the intercom device 1100 applied to the kitchen environment may further include: a business feature extraction module and a knowledge matching module.

[0265] The business feature extraction module can be used to extract kitchen business features from the monitored content; the knowledge matching module can be used to perform knowledge matching between the kitchen business features and vector data in the preset kitchen knowledge base, so as to generate an audio stream that conforms to the kitchen business scenario based on the matched preset kitchen knowledge constraint big language model.

[0266] In some embodiments, the intercom device 1100 applied to the kitchen environment may further include: a second intercom voice stream production module.

[0267] The second intercom voice stream production module can be used to generate a second intercom audio stream through at least one of the following methods, and push the second intercom audio stream to the kitchen monitoring terminal through the voice backhaul link: detecting kitchen violations in the monitored content; if a preset violation feature is detected, determining a preset alarm text corresponding to the violation feature; converting the preset alarm text into the second intercom audio stream and pushing it to the kitchen monitoring terminal; acquiring structured kitchen order data from the kitchen business system; parsing the structured kitchen order data based on preset trigger conditions to generate an order prompt text instruction; inputting the order prompt text instruction into a speech synthesis model to generate the second intercom audio stream.

[0268] Since the functions of the device 1100 have been described in detail in their respective method embodiments, they will not be repeated here.

[0269] The modules (and / or sub-modules and / or) units described in the embodiments of this disclosure can be implemented in software or in hardware. The described modules (and / or sub-modules and / or) units can also be located in a processor. The names of these modules (and / or sub-modules and / or) units do not, in some cases, constitute a limitation on the module (and / or sub-module and / or) unit itself.

[0270] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module or program segment containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer program instructions.

[0271] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0272] Figure 12A schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 12 The illustrated electronic device 1200 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0273] like Figure 12 As shown, the electronic device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage section 1208 into a random access memory (RAM) 1203. The RAM 1203 also stores various programs and data required for the operation of the electronic device 1200. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0274] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0275] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing computer program instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs the functions defined above in the system of this disclosure.

[0276] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable computer program instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Computer program instructions contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0277] In another aspect, this disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable storage medium carries one or more programs that, when executed by the device, enable the device to perform the following functions: receiving a message sequence from a kitchen monitoring terminal; wherein the message sequence contains payload data using an MPEG-2 program stream encapsulation format; based on the encapsulation header identifier in the payload data, converting the payload data into a distribution data stream conforming to a streaming media protocol and distributing it to a client, the client being used to play video footage and audio of the kitchen environment; acquiring an intercom audio stream for the monitoring content in the payload data; wherein the encoding format of the intercom audio stream matches the audio encoding format in the payload data; and, in response to an intercom command sent by the client, pushing the intercom audio stream to the kitchen monitoring terminal through a voice feedback link established with the kitchen monitoring terminal.

[0278] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0279] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive) and includes several computer program instructions to cause an electronic device (such as a server or terminal device) to execute the method according to the embodiments of this disclosure.

[0280] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0281] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for intercom use in a kitchen environment, characterized in that, include: Receive a message sequence from the kitchen monitoring terminal; wherein the message sequence contains payload data in MPEG-2 program stream encapsulation format; Based on the encapsulation header identifier in the payload data, the payload data is converted into a distribution data stream conforming to the streaming media protocol and distributed to the client, which is used to play the video footage and audio of the kitchen environment; Acquire an intercom audio stream for the monitored content in the payload data; wherein the encoding format of the intercom audio stream matches the audio encoding format in the payload data; In response to the intercom command sent by the client, the intercom audio stream is pushed to the kitchen monitoring terminal through the voice feedback link established with the kitchen monitoring terminal.

2. The method according to claim 1, characterized in that, The method further includes: In response to the intercom command, an idle port is dynamically allocated from the pre-configured port pool as the server-side listening port for the voice backhaul link. The address information of the server listening port is sent to the kitchen monitoring terminal so that the kitchen monitoring terminal can initiate a connection request to the server listening port and establish the voice transmission link.

3. The method according to claim 1, characterized in that, The method further includes: During the process of pushing the intercom audio stream to the kitchen monitoring terminal, the real-time audio stream in the monitoring content is detected in real time. In response to the detection of new voice activity from the real-time audio stream, an interruption event is triggered; In response to the interruption event, perform at least one of the following operations: The transmission of the intercom audio stream to the kitchen monitoring terminal is interrupted. Clear any unsent audio frames from the audio buffer; Begin acquiring and pushing the new intercom audio stream corresponding to the new voice activity.

4. The method according to claim 1, characterized in that, Pushing the intercom audio stream to the kitchen monitoring terminal includes: Obtain the current system time and the start time of pushing the intercom audio stream to the kitchen monitoring terminal; Determine the difference between the current system time and the initial streaming system time; Based on the starting push system time and the difference, the audio index of the audio frame that should be sent in the intercom audio stream is determined as the first audio frame index; The audio frame corresponding to the first audio frame index is extracted from the audio buffer and pushed to the kitchen monitoring terminal.

5. The method according to claim 1, characterized in that, Pushing the intercom audio stream to the kitchen monitoring terminal includes: Obtain the first video timestamp corresponding to the monitoring content played by the client when the intercom is triggered; Obtain the second video timestamp corresponding to the monitoring content currently being played on the client; Calculate the time offset between the second video timestamp and the first video timestamp; Based on the time offset, a target audio frame corresponding to the time offset is determined from the intercom audio stream, wherein the start playback time of the intercom audio stream corresponds to the first video timestamp; The target audio frame is extracted from the audio buffer and associated with the second video timestamp before being pushed to the kitchen monitoring terminal, so that the kitchen monitoring terminal can synchronously play the target audio frame when playing the monitoring video screen corresponding to the second video timestamp.

6. The method according to claim 1, characterized in that, Acquiring the intercom audio stream for the monitored content in the payload data includes: In response to the voice intercom request initiated by the client, an uplink long connection is established with the client using the WebSocket long connection protocol; The uplink long connection receives an intercom audio stream from the client for the monitored content in the payload data.

7. The method according to claim 1, characterized in that, The method further includes: The payload data is subjected to audio retrieval and audio format conversion to obtain the target audio; The target audio is subjected to speech recognition to obtain the recognized text information; The identified text information is input into a large language model so that the large language model can generate a response text based on the identified text information. A first prompt audio stream is synthesized based on the reply text; The first prompt audio stream is pushed to the kitchen monitoring terminal via the voice feedback link.

8. The method according to claim 1, characterized in that, The method further includes: Extract kitchen business characteristics from the monitored content; The kitchen business characteristics are matched with vector data in a preset kitchen knowledge base to generate an audio stream that conforms to the kitchen business scenario based on the matched preset kitchen knowledge constraint big language model.

9. The method according to claim 1, characterized in that, The method further includes generating a second intercom audio stream by at least one of the following methods, and pushing the second intercom audio stream to the kitchen monitoring terminal via the voice return link: The monitored content is used to detect kitchen violations; if a preset violation feature is detected, the preset alarm text corresponding to the violation feature is determined. The preset alarm text is converted into the second intercom audio stream and pushed to the kitchen monitoring terminal; Obtain structured kitchen order data from the kitchen operations system; Based on preset triggering conditions, the structured kitchen order data is parsed to generate order prompt text instructions; The order prompt text instruction is input into the speech synthesis model to generate the second intercom audio stream.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the intercom method applied to the kitchen environment as described in any one of claims 1-9.

Images