A mobile information confirmation interaction method to prevent misoperation

Abstract

This invention discloses a mobile information anti-misoperation confirmation interaction method, including the following steps: T-1, Signaling Registration: When the client first accesses the platform, it continuously sends REGISTER messages until the server responds with "200 OK" indicating successful registration. If it fails, it will randomly delay and retry registration. Simultaneously, the client periodically refreshes the registration. T-2, Data Transmission Interaction: After successful signaling registration, the server sends directory information to the client. The client constructs the content and sends it to the server. The server verifies the content and returns an INVITE response. Employing a "finger-based verbal communication" technology solution, the on-site situation can be transmitted in high definition to remote experts in real time for consultation. Experts can provide real-time multi-party high-definition video guidance to on-site personnel, whether on a PC or smartphone. Tools such as screen freeze, whiteboard, file transfer, and screen sharing are used to collaboratively solve on-site problems using the finger-based verbal communication technology solution.

Description

Technical Field

[0001] The present invention relates to the technical field of visual confirmation interaction, and in particular to a mobile information confirmation interaction method for preventing misoperation in the petrochemical industry. Background Art

[0002] The petrochemical industry is characterized by high temperature, high pressure, flammability, explosiveness, toxicity and harm. The amount of on-site construction work is large. If it is not properly managed and controlled, safety accidents such as poisoning, suffocation, fire and explosion are very likely to occur, which will not only have a serious impact on corporate interests, but also seriously threaten people's lives and safety.

[0003] At present, security monitoring in fixed scenarios in the petrochemical industry has been relatively complete, but the monitoring of operation processes in mobile operations needs to be urgently addressed. According to relevant requirements, the core of finger dictation is to provide a technical solution to solve the problem of mobile operation monitoring.

[0004] Finger dictation uses audio and video technology to guide on-site operations in the petrochemical industry. Its main purpose is to ensure safe operations. The core solution of this technology is to transmit the on-site situation in high definition to remote experts for consultation. Experts can provide real-time multi-party high-definition video guidance with on-site personnel on both PCs and smartphones, and use tools such as frozen screen, whiteboard, file transfer, and shared screen to jointly solve problems encountered on-site. This invention has been promoted and applied throughout Changling Petrochemical.

[0005] The existing technical problems are: Mobile 4 / 5G surveillance ball: limited field of view, lack of flexibility, difficulty in tracking moving targets, limited viewing angle, easy to be blocked, the effect cannot be guaranteed under bad weather conditions, it is not easy to obtain video information on site in the first time, and there is no interactive capability.

[0006] Legal recorder: obvious jitter, difficult to obtain long-term stable images, the lens is not synchronized with the line of sight, the recorded content may not be what is needed, the lens is easily blocked by clothing, limbs or equipment, etc., high stability requirements for the wearer, easy to distort when moving, difficult to obtain video information on the scene in the first time, no interactive ability, for this reason, we propose a mobile information to prevent misoperation confirmation interaction method. Summary of the invention

[0007] In view of this, the embodiment of the present invention hopes to provide a mobile information confirmation interaction method for preventing misoperation, the purpose of which is to interact with information during mobile operations, and remote experts can perform finger-dictated operations on on-site operators, and at least provide a beneficial option for this;

[0008] The technical solution of the present invention is achieved in this way:

[0009] A mobile information confirmation interaction method to prevent misoperation. During the mobile operation process, information is exchanged and remote experts can perform finger-dictated operations on on-site workers.

[0010] 1. Overview

[0011] This application aims to further develop technical solutions for information interaction during mobile operations. By collecting audio and video data through smart devices such as head-mounted computers, the audio and video data are processed and converted into video streams that comply with the national standard GB / T28181 protocol, which can be pushed to the finger-dictation visualization management platform. At the same time, the finger-dictation platform can push the audio and video streams to smart devices such as head-mounted computers in real time to achieve audio interactive communication.

[0012] "Finger-to-mouth" is a method of safety confirmation, which is mainly used in jobs and industries with higher risks, mainly in mining and other industries.

[0013] (II) Technical content

[0014] A first aspect: A mobile information confirmation interaction method for preventing misoperation, comprising the following steps T-1, T-2, T-3 and T-4:

[0015] T-1, signaling registration: When the client accesses the platform for the first time, it will continue to send REGISTER messages until the server responds with "200 OK" to indicate successful registration. If it fails, it will try to register again after a random delay. At the same time, the client will refresh the registration periodically;

[0016] T-2, data transmission interaction: After the signaling registration is successful, the server will send the directory information to the client, the client will build the content and send it to the server, the server will verify it and return the INVITE response;

[0017] T-3, obtaining client audio and video codec data; wherein, extracting associated information of the audio and video codec data, the associated information includes:

[0018] S3.1, Video encoding: The data collected by the camera is encoded into h264 through MediaCodec, and then packaged into PS using RTP protocol for transmission;

[0019] S3.2, Audio encoding: AudioRecord is used to collect sound, encode it in G711, and then transmit it in RTP packaging;

[0020] S3.3, mixed stream encoding: Mix the audio and video streams through MediaMuxer, initialize MediaExtractor to extract video and audio track information from video files and audio files and obtain their formats;

[0021] The T-4, obtains the audio media stream interaction of the cloud platform: the client and the cloud platform establish a broadcast connection through signaling, and finally realizes audio and video interaction.

[0022] Preferably: in the T-1, the signaling registration specifically includes:

[0023] 1) The client initiates a registration request to the SIP server. After receiving the request, the server verifies the registration information and sends a response to the client after the verification passes. The client's authentication system and parameters are given in the header WWW-Authenticate field of the response.

[0024] 2) After receiving the authentication system and parameter data, the client resends a registration request to the SIP server, providing a letter of trust and carrying authentication information in the Authorization field of the request;

[0025] 3) The SIP server receives the request and verifies it again. After verification, if the client identity is legal, it sends a successful response "200 OK" to the client. If the client identity is illegal, it sends a service denial response message.

[0026] Preferably, in said T-2, said data transmission interaction: the specific steps include:

[0027] S2.1. After the signaling registration is successful, the server will send the CATALOG directory information to the client;

[0028] S2.2, the client receives the CATALOG message and constructs the content, and sends it to the server after completion;

[0029] S2.3. After receiving the CATALOG information, the server verifies the content. After the verification passes, the server sends an INVITE response to the client.

[0030] S2.4. After receiving the INVITE response, the client encapsulates the INVITE information and SDP information and responds to the server. The client obtains the RTP transmission port according to the INVITE information and SDP information sent by the server, wherein the SDP information includes TCP, UDP mode and SSRC information.

[0031] S2.5. After all connections are established, the client packages the collected audio and video data into a PS stream that complies with the national standard GB / T28181 protocol and sends it to the server in the form of RTP packets;

[0032] For the PS stream mode, a data stream packaging format commonly used in the national standard GB / T28181 protocol, the field describing the stream structure in the video attribute field set in the SDP file is written as: a=rtpmap:96MP2P / 90000, a=rtpmap is a fixed format, 96 is the payload type, MP2P is the encoding type, and 90000 is the clock frequency; following the previous line a=rtpmap:96MP2P / 90000, a=fmtp:96 is designed, where a=fmtp is a fixed format and 96 is the payload type;

[0033] S2.6. If the client needs to disconnect the Invite session, it will need to send a BYE request to the server. The server will disconnect after returning "200 OK";

[0034] S2.7. If the client initiates a logout: it sends a Register command message to the server, and the Expire field in the message is set to 0, which means logout, and the session is terminated at this time.

[0035] Preferably, in S3.1, the video encoding: encoding the data collected by the camera into h264 through MediaCodec, and then using the RTP protocol to package PS for transmission, specifically includes:

[0036] S3.11, initialize and obtain the original data returned by the device Camera2 through PreviewFrame real-time monitoring detection camera collection;

[0037] S3.12, convert the collected NV21 data into h264 encoding into YUV420P data;

[0038] S3.13. Create a MediaCodec instance to set encoding format parameters and perform h264 hard encoding on YUV data;

[0039] S3.14, obtaining the encoded h264 data, and encapsulating the h264 data in PS format according to the national standard GB / T28181 protocol;

[0040] S3.15. Send the encapsulated data to the server based on RTP / UDP.

[0041] Preferably, in S3.2, the audio encoding: collecting sound through AudioRecord and performing G711 encoding RTP packaging transmission and sending, specifically includes:

[0042] S3.21, initialize AudioRecord to set encoding format parameters and collect audio data;

[0043] S3.22, after starting recording, write the acquired audio data into the created byte buffer through audioRecord.read;

[0044] S3.23, read the data in the buffer area and perform short type conversion, and encode the converted short data into G711;

[0045] S3.24. The encoded data is packaged and sent to the server via the RTP protocol.

[0046] Preferably, in S3.3, the mixed stream encoding: mixing the audio and video streams through MediaMuxer, initializing MediaExtractor to extract video and audio track information from the video file and the audio file and obtain their formats, specifically includes:

[0047] S3.31, initialize MediaExtractor to extract video and audio track information from video files and audio files;

[0048] S3.32, use MediaExtractor.readSampleData to read one frame of data from the video file and audio file respectively into the buffer area;

[0049] S3.33, instantiate the MediaMuxer object to merge the audio and video data in the buffer and output them;

[0050] S3.34, set the buffer information, including the timestamp and flag of the current frame, to determine whether it is a key frame;

[0051] S3.35. Send using DatagramSocket based on RTP / UDP.

[0052] Preferably, in said T-4, the audio media stream interaction of the cloud platform is obtained:

[0053] S4.1. The cloud platform sends a MESSAGE voice broadcast command to the client;

[0054] S4.2, the client sends a MESSAGE message in response to the voice broadcast command;

[0055] S4.3. After receiving the MESSAGE response "200 OK", the client sends an INVITE message:

[0056] S4.4, the client receives the INVITE and the "200 OK" response;

[0057] S4.5. After the client replies with an ACK message, it starts to read the RTP data packet, parse the RTP data packet, run the encoding logic to decode the audio data, and output it to the client for playback, thus completing the interaction.

[0058] Cloud platforms include: Uniview platform or Hikvision platform.

[0059] Preferably, the client includes an intelligent head-mounted computer, a microphone, a control camera, a display optical machine and an intelligent glasses processor.

[0060] Preferably: in said T-2, said data transmission interaction further includes: a data source, a device access terminal and an international platform terminal;

[0061] The data source includes data before external encoding, Android camera (including Android Camera2), screen data, encoded data H264, HEVC, RTSP or RTMP stream data, and the data type before external encoding is any one of YV12, NV21, NV12, 1420, RGB24, RGBA32, GB565;

[0062] The device access terminal is used for national standard GB / T 28181 device access signaling registration, streaming media transmission, and voice broadcast reception;

[0063] The international platform is the national standard GB / T 28181 platform;

[0064] The data source is connected to the device access terminal, and the device access terminal is interactively connected to the international platform terminal.

[0065] The second aspect: a mobile information confirmation interactive system to prevent misoperation;

[0066] The system includes a processor and a memory coupled to the processor, wherein program instructions are stored in the memory, and when the program instructions are executed by the processor, the processor executes the mobile information misoperation prevention confirmation interaction method as described above.

[0067] (1) Processor: The processor is the core component of the interactive system and is responsible for executing each step in the mobile information error prevention confirmation interactive method. It can process input data and perform corresponding operations according to program instructions, thereby realizing the operation of the entire interactive method.

[0068] (2) Memory: The memory is coupled to the processor and is used to store program instructions. These program instructions are the key to implementing the mobile information error-prevention confirmation interaction method and can guide the processor to perform calculations and processing according to pre-designed steps.

[0069] (3) Program instruction execution: Program instructions are stored in the memory. When the processor executes these instructions, it will calculate and process data according to the pre-designed steps and algorithms to implement each stage of the mobile information error prevention confirmation interaction method. These instructions will guide the processor to complete the entire mobile information error prevention confirmation interaction method.

[0070] In summary, compared with the prior art, the mobile information error-prevention confirmation interaction method provided by the present invention has the following beneficial effects:

[0071] (1) The “finger-to-mouth” technical solution can transmit the on-site situation in high definition to remote experts in real time for consultation. Experts can provide real-time multi-party high-definition video guidance with on-site personnel on either PC or smartphone, and use tools such as freezing the screen, whiteboard, file transfer, and screen sharing to jointly solve problems encountered on-site using the finger-to-mouth technical solution.

[0072] (2) The mobile operation information interaction method is adopted to solve the information interaction problem during the operation process. The entire operation process is recorded to improve operation safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 It is a logical schematic diagram of the method flow of the present invention;

[0075] Figure 2 A schematic diagram of the interaction logic between the SIP server and the SIP proxy of the present invention;

[0076] Figure 3 This is a schematic diagram of the data transmission interaction process of the third embodiment of the method of the present invention;

[0077] Figure 4 This is a data transmission interaction logic block diagram of the method embodiment 3 of the present invention;

[0078] Figure 5 This is a flow chart of the fourth embodiment of the method of the present invention;

[0079] Figure 6 This is a flow chart of the fifth embodiment of the method of the present invention;

[0080] Figure 7 This is a flow chart of the sixth embodiment of the method of the present invention;

[0081] Figure 8 This is a flow chart of the logic of the seventh embodiment of the method of the present invention. DETAILED DESCRIPTION

[0082] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below;

[0083] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0084] For those skilled in the art, it can be further appreciated that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0085] It should be noted that the steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software modules executed by a processor, or a combination of the two. The software modules may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0086] Example 1: Please refer to Figure 1 This embodiment provides a mobile information error-prevention confirmation interaction method, including the following steps T-1, T-2, T-3 and T-4:

[0087] T-1, signaling registration: When the client accesses the platform for the first time, it will continue to send REGISTER messages until the server responds with "200 OK" to indicate successful registration. If it fails, it will try to register again after a random delay. At the same time, the client will refresh the registration periodically;

[0088] T-2, data transmission interaction: After the signaling registration is successful, the server will send the directory information to the client, the client will build the content and send it to the server, the server will verify it and return the INVITE response;

[0089] T-3, obtaining client audio and video codec data; wherein, extracting associated information of the audio and video codec data, the associated information includes:

[0090] S3.1, Video encoding: The data collected by the camera is encoded into h264 through MediaCodec, and then packaged into PS using RTP protocol for transmission;

[0091] S3.2, Audio encoding: AudioRecord is used to collect sound, encode it in G711, and then transmit it in RTP packaging;

[0092] S3.3, mixed stream encoding: Mix the audio and video streams through MediaMuxer, initialize MediaExtractor to extract video and audio track information from video files and audio files and obtain their formats;

[0093] The T-4, obtains the audio media stream interaction of the cloud platform: the client and the cloud platform establish a broadcast connection through signaling, and finally realizes audio and video interaction.

[0094] In this embodiment, regarding T-1: signaling registration: when the client first accesses the platform, it will continue to send REGISTER messages until the server responds with "200 OK" indicating successful registration. If it fails, it will retry registration after a random delay. At the same time, the client will refresh the registration periodically. For the specific implementation of T-1, please refer to Example 2.

[0095] See also Figure 2 Specifically, in this embodiment, taking basic registration as an example: basic registration adopts the digital summary-based challenge-response security technology specified in IETFRFC3261 for registration.

[0096] Specifically, the client includes an intelligent head-mounted computer, a microphone, a control camera, a display optical machine and an intelligent glasses processor.

[0097] Specifically, when the smart head-mounted computer device is connected to the platform for the first time, the device will continue to send REGISTER messages to the platform until the server replies "200 OK" to indicate successful registration. If the device or system registration is unsuccessful, it is recommended to re-register after a certain random delay. The client will refresh the registration with the server again at regular intervals to prevent the connection from being disconnected due to registration expiration.

[0098] In this embodiment, regarding T-2: Data transmission interaction: After the signaling registration is successful, the server will send directory information to the client, the client will build the content and send it to the server, the server will verify and return the INVITE response. For the specific implementation of T-2, please refer to Example 3.

[0099] In this embodiment, regarding T-3: obtaining audio and video codec data from the client; wherein, extracting associated information of the audio and video codec data, the associated information includes:

[0100] S3.1, Video encoding: The data collected by the camera is encoded into h264 through MediaCodec, and then packaged into PS using RTP protocol for transmission;

[0101] S3.2, Audio encoding: AudioRecord is used to collect sound, encode it in G711, and then transmit it in RTP packaging;

[0102] S3.3, mixed stream encoding: Mix the audio and video streams through MediaMuxer, and initialize MediaExtractor to extract video and audio track information from the video file and audio file and obtain their formats. For the specific implementation of T-3, please refer to Example 4, Example 5 and Example 6.

[0103] In this embodiment, regarding T-4: obtaining audio media stream interaction of the cloud platform: the client and the cloud platform establish a broadcast connection through signaling, and finally realize audio and video interaction. For the specific implementation of T-4, please refer to Example 7.

[0104] In summary, the "finger dictation" technical solution can transmit the on-site situation in real time and in high definition to remote experts for consultation. Experts can provide real-time multi-party high-definition video guidance with on-site personnel on both PCs and smartphones, and use tools such as freezing the screen, whiteboards, file transfer, and sharing the screen, and use the finger dictation technical solution to jointly solve problems encountered on-site.

[0105] Example 2: Please refer to Figure 2 Based on Example 1, this example will further disclose a specific implementation of T-1: In T-1, it includes:

[0106] 1) The client initiates a registration request to the SIP server. After receiving the request, the server verifies the registration information. After the verification is passed, it sends a 401 response to the client and gives the client's authentication system and parameters in the header WWW-Authenticate field of the response.

[0107] 2) After receiving the authentication system and parameter data, the client resends a registration request to the SIP server, providing a letter of trust and carrying authentication information in the Authorization field of the request;

[0108] 3) The SIP server receives the request and verifies it again. After verification, if the client identity is legal, it sends a successful response "200 OK" to the client. If the client identity is illegal, it sends a service denial response message.

[0109] Specific: Features of the specific implementation of T-1 are disclosed:

[0110] 1) Improved security: Through multiple identity authentication and information verification, the SIP server improves the degree of confirmation of the requester's identity, making it more difficult for illegal users to pass the verification.

[0111] 2) Two-way interaction between client and server: The communication between client and server is not just one-way. The server first sends the authentication system and parameters to the client, and then the client responds based on this information. This two-way interaction improves the degree of information synchronization between the server and the client.

[0112] 3) Enhanced system robustness: When receiving a request with an illegal identity, the server will not attempt to process the erroneous request, but will directly return a response message of service rejection, avoiding possible error handling and ensuring stable system operation.

[0113] 4) Flexible response mode: The client responds in a corresponding manner based on the server's response and the parameter information it carries. If the response is positive, the server needs to send a successful response "200 OK" to the client, making the whole process more flexible.

[0114] In summary, the specific implementation of T-1 in this embodiment makes the verification process more flexible and greatly improves the security and system robustness.

[0115] Example 3: Please refer to Figure 3 In this embodiment, based on the first embodiment, the specific implementation of T-2 will be further disclosed:

[0116] The data transmission interaction: the specific steps include:

[0117] S2.1. After the signaling registration is successful, the server will send the CATALOG directory information to the client;

[0118] S2.2, the client receives the CATALOG message and constructs the content, and sends it to the server after completion;

[0119] S2.3. After receiving the CATALOG information, the server verifies the content. After the verification passes, the server sends an INVITE response to the client.

[0120] S2.4. After receiving the INVITE response, the client encapsulates the INVITE information and SDP information and responds to the server. The client obtains the RTP transmission port according to the INVITE information and SDP information sent by the server, wherein the SDP information includes TCP, UDP mode and SSRC information.

[0121] S2.5. After all connections are established, the client packages the collected audio and video data into a PS stream that complies with the national standard GB / T28181 protocol and sends it to the server in the form of RTP packets;

[0122] For the PS stream mode, a data stream packaging format commonly used in the national standard GB / T28181 protocol, the field describing the stream structure in the video attribute field set in the SDP file is written as: a=rtpmap:96MP2P / 90000, a=rtpmap is a fixed format, 96 is the payload type, MP2P is the encoding type, and 90000 is the clock frequency; following the previous line a=rtpmap:96MP2P / 90000, a=fmtp:96 is designed, where a=fmtp is a fixed format and 96 is the payload type;

[0123] During the transmission process, the client must intermittently send KEEP ALIVE heartbeat packets to the server to maintain a long connection and avoid disconnection that causes data transmission failure.

[0124] S2.6. If the client needs to disconnect the Invite session, it will need to send a BYE request to the server. The server will disconnect after returning "200 OK";

[0125] S2.7. If the client initiates a logout: it sends a Register command message to the server, and the Expire field in the message is set to 0, which means logout, and the session is terminated at this time.

[0126] Specifically, the complete data transmission and interaction process is safe, controllable and efficient:

[0127] 1) Security: Before the connection is established, the server performs multiple rounds of verification on the client's request information, which helps prevent access by illegal devices and ensures secure data transmission.

[0128] 2) Controllability: Through the SIP protocol, the client and the server can perform detailed session control, including session establishment, modification, and termination, meeting the real-time and controllability requirements of communication.

[0129] 3) Efficiency: All audio and video data will be packaged into PS streams that comply with the national standard GB / T28181 protocol before transmission, and sent in the form of RTP packets, ensuring efficient and stable data transmission. The fields of the stream structure are described in detail in the SDP file, providing a clear data description to facilitate subsequent data processing.

[0130] 4) Flexibility: When the client needs to disconnect or log out, corresponding instructions can be sent, making the whole process more flexible and practical.

[0131] See also Figure 4 , specifically: in said T-2, said data transmission interaction also includes: a data source, a device access terminal and an international platform terminal;

[0132] The data source includes data before external encoding, Android camera (including Android Camera2), screen data, encoded data H264, HEVC, RTSP or RTMP stream data, and the data type before external encoding is any one of YV12, NV21, NV12, 1420, RGB24, RGBA32, GB565;

[0133] The device access terminal is used for national standard GB / T 28181 device access signaling registration, streaming media transmission, and voice broadcast reception;

[0134] The international platform is the national standard GB / T 28181 platform;

[0135] The data source is connected to the device access terminal, and the device access terminal is interactively connected to the international platform terminal.

[0136] In summary, the specific implementation of T-2 in this embodiment makes the entire data transmission process have good security, controllability and efficiency, and can flexibly respond to different usage requirements.

[0137] Example 4: Please refer to Figure 5In this embodiment, based on the first embodiment, this embodiment will further disclose the video encoding in S3.1: the data collected by the camera is encoded into h264 by MediaCodec, and then packaged into PS for transmission using the RTP protocol. The specific implementation method is as follows:

[0138] S3.11, initialize and obtain the original data returned by the device Camera2 through PreviewFrame real-time monitoring detection camera collection;

[0139] S3.12, convert the collected NV21 data into h264 encoding into YUV420P data;

[0140] S3.13. Create a MediaCodec instance to set encoding format parameters and perform h264 hard encoding on YUV data;

[0141] S3.14, obtaining the encoded h264 data, and encapsulating the h264 data in PS format according to the national standard GB / T28181 protocol;

[0142] S3.15. Send the encapsulated data to the server based on RTP / UDP.

[0143] In the specific example, initialize and obtain the device Camera2 instance, call the ImageReader.newInstance method to obtain the original data stream returned by the camera preview; create a MediaCodec instance, set the encoding format and other parameters to encode the camera raw data, and when the camera is turned on, continuously obtain the original YUV data by implementing the ImageReader.OnImageAvailableListener callback, create a byte array to buffer the data to avoid blockage and data callback termination, call MediaCodec to encode the YUV data in h264, and when the encoding and decoding operation starts, start the encoding and decoding thread, process the YUV data returned by the camera preview, and obtain the index of the available input buffer through the dequeueInputBuffer method;

[0144] Call getInputBuffer to get the input buffer, write the data into the buffer, and queue it through the queueInputBuffer method after filling the data. Call queueInputBuffer to submit the inputBuffer filled with data to the encoding queue for encoding. After the encoding is completed, call dequeueOutputBuffer to obtain the index of the output buffer that has been successfully encoded and decoded. Finally, get the output buffer to get the encoded h264 data. After completion, release the buffer, package the h264 data into PS stream, and send it according to the RTP protocol.

[0145] In summary, the specific implementation of S3.1 in this embodiment has the following features:

[0146] 1) High efficiency: Directly performing h264 hard encoding processing through MediaCodec avoids the complex calculation process of soft encoding and greatly improves the encoding efficiency.

[0147] 2) Good real-time performance: Get the original data of the device Camera, and monitor it in real time through PreviewFrame, so as to quickly obtain and convert the data returned by the camera.

[0148] 3) Strong compatibility: The collected NV21 data is encoded into YUV420P data using h264, and encapsulated in the PS format of the widely accepted national standard GB / T28181 protocol, which not only improves transmission efficiency but also enhances compatibility between devices.

[0149] 4) Easy to transmit: Sending encapsulated data based on RTP / UDP can ensure smooth transmission of data in the network, making data transmission more smooth and ensuring the continuity of video in the event of data loss.

[0150] 5) Strong adaptability: Because the video data is packaged as a PS stream, it means that a series of audio and video encodings can be packaged, and audio and video data in various formats can be flexibly processed.

[0151] Example 5: Please refer to Figure 6 In this embodiment, based on the first embodiment, the audio encoding in S3.2 is further disclosed: the audio is collected through AudioRecord for G711 encoding and RTP packaging transmission and transmission. The specific implementation method is:

[0152] S3.21, initialize AudioRecord to set encoding format parameters and collect audio data;

[0153] S3.22, after starting recording, write the acquired audio data into the created byte buffer through audioRecord.read;

[0154] S3.23, read the data in the buffer area and perform short type conversion, and encode the converted short data into G711;

[0155] S3.24. The encoded data is packaged and sent to the server via the RTP protocol.

[0156] Specifically, initialize AudioRecord to set parameters such as the encoding format to collect audio data. After starting recording, use audioRecord.read to monitor in real time whether there is audio data. When there is data, write the data to the created byte buffer. If the buffer length is greater than 0, read the data in the buffer and convert it to short type. Perform G711 encoding on the converted short data. After encoding, package it and send it to the server through the RTP protocol.

[0157] In summary, the specific implementation of S3.2 in this embodiment has the following characteristics:

[0158] 1) High efficiency: Audio data is directly obtained through AudioRecord technology and G711 encoding is performed. Compared with other encoding methods, G711 encoding can provide higher audio quality and has low hardware requirements.

[0159] 2) Good quality: Using the RTP protocol to encapsulate and transmit audio data can ensure the smooth transmission of audio data in the network and reduce the impact of data loss on audio quality.

[0160] 3) Strong real-time performance: The solution reads and encodes data during recording, realizing real-time transmission of audio.

[0161] 4) Simple operation: Audio data can be directly read through audioRecord.read, and then only simple short type conversion and encoding of the audio data is required, which reduces the complexity of operation.

[0162] 5) Good compatibility: Because it adopts the internationally accepted G711 audio encoding method and RTP protocol, the solution has good device compatibility and system compatibility.

[0163] The specific implementation of S3.2 makes the audio encoding and transmission process highly real-time, efficient and compatible.

[0164] Example 6: Please refer to Figure 7In this embodiment, based on the first embodiment, the mixed stream encoding in S3.3 is further disclosed: the audio and video are mixed by MediaMuxer, and MediaExtractor is initialized to extract video and audio track information from the video file and the audio file and obtain their formats;

[0165] S3.31, initialize MediaExtractor to extract video and audio track information from video files and audio files;

[0166] S3.32, use MediaExtractor.readSampleData to read one frame of data from the video file and audio file respectively into the buffer area;

[0167] S3.33, instantiate the MediaMuxer object to merge the audio and video data in the buffer and output them;

[0168] S3.34, set the buffer information, including the timestamp and flag of the current frame, to determine whether it is a key frame;

[0169] S3.35. Send using DatagramSocket based on RTP / UDP.

[0170] Specifically, use MediaExtractor.readSampleData to read one frame of data from the video file and audio file into the buffer. When the data is greater than 0, instantiate the MediaMuxer object to merge the audio and video data in the buffer and output it. Set the buffer information during output, including the timestamp of the current frame and whether the flag is a key frame, etc. Then move to the next frame for merging and use DatagramSocket.send to send it.

[0171] In summary, in S3.3, the specific implementation method, the characteristics of the mixed stream encoding are:

[0172] 1) Efficiency: By using MediaExtractor and MediaMuxer, audio and video data can be directly extracted from the original files and mixed without decoding and re-encoding, which greatly improves processing efficiency.

[0173] 2) Flexibility: The solution provides detailed settings for each frame of data, including timestamps and key frame flags, which not only facilitates subsequent data processing and decoding, but also makes the control of the video stream more flexible.

[0174] 3) Easy operation: You only need to initialize the corresponding objects to read and mix audio and video data. The operation is simple and easy, and no complicated data processing steps are required.

[0175] 4) Strong compatibility: It uses the widely accepted RTP / UDP protocol for data transmission, and is sent based on DatagramSocket, which has good compatibility with various devices and systems.

[0176] 5) Strong scalability: Based on this solution, it can be easily expanded, such as adding new audio tracks, modifying video quality, etc.

[0177] Example 7: Please refer to Figure 8 In this embodiment, based on the first embodiment, T-4: obtaining audio media stream interaction of the cloud platform: the client and the cloud platform establish a broadcast connection through signaling, and finally realize the specific implementation method of audio and video interaction will be further disclosed:

[0178] The audio media stream interaction of the cloud platform is obtained:

[0179] S4.1. The cloud platform sends a MESSAGE voice broadcast command to the client;

[0180] S4.2, the client sends a MESSAGE message in response to the voice broadcast command;

[0181] S4.3. After receiving the MESSAGE response "200 OK", the client sends an INVITE message:

[0182] S4.4, the client receives the INVITE and the "200 OK" response;

[0183] S4.5. After the client replies with an ACK message, it starts to read the RTP data packet, parse the RTP data packet, run the encoding logic to decode the audio data, and output it to the client for playback, thus completing the interaction.

[0184] Specifically, the features of the specific implementation of T-4 are disclosed:

[0185] 1) Effectiveness: Through a clear command response process, the communication between the cloud platform and the client is carried out effectively. The cloud platform issues commands, the client responds to the commands, and takes the next step based on the received response, which ensures the effectiveness of the entire audio media stream acquisition process.

[0186] 2) Real-time: During the entire interaction process, the client can respond to the cloud platform's instructions in real time and perform the next operation, which ensures the real-time interaction of the audio media stream of the cloud platform.

[0187] 3) Stability: After receiving the "200 OK" response, the client replies with an ACK message and performs subsequent processing. This process reflects a stable communication process. Even in the event of network fluctuations or other situations, the reliability of communication is guaranteed by replying with an ACK message.

[0188] 4) Easy to decode: After receiving the RTP data packet, the client parses the data packet and runs the encoding logic to decode the audio data, which allows the audio data to be played smoothly on the client and complete the interaction.

[0189] Embodiment 8: This embodiment further discloses a mobile information error-prevention operation confirmation interaction system; the system includes a processor and a memory coupled to the processor, the memory stores program instructions, and when the program instructions are executed by the processor, the processor executes the mobile information error-prevention operation confirmation interaction method as described above. Specifically:

[0190] The memory stores program instructions, and when the program instructions are executed by the processor, the processor executes the steps of the mobile information prevention misoperation confirmation interaction method in the above embodiment.

[0191] Processor: The processor is the core component of the interactive system and is responsible for executing each step in the mobile information to prevent misoperation confirmation interactive method. It can process input data and perform corresponding operations according to program instructions, thereby realizing the operation of the entire interactive method.

[0192] The processor may also be referred to as a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0193] (2) Memory: The memory is coupled to the processor and is used to store program instructions. These program instructions are the key to implementing the mobile information error-prevention confirmation interaction method and can guide the processor to perform calculations and processing according to pre-designed steps.

[0194] (3) Program instruction execution: Program instructions are stored in the memory. When the processor executes these instructions, it will calculate and process data according to the pre-designed steps and algorithms to implement each stage of the mobile information error prevention confirmation interaction method. These instructions will guide the processor to complete the entire mobile information error prevention confirmation interaction method based on cellular automation.

[0195] The storage medium of the embodiment of the present application stores program instructions, and the program instructions are executed by the processor to implement the mobile information prevention misoperation confirmation interaction method in the above embodiment, wherein the computer program can be stored in the above storage medium in the form of a software product, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a computer device such as a computer, a server, a mobile phone, and a tablet. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0196] (4) System workflow:

[0197] P1. Input data import: Obtain the client audio and video codec data as input and import it into the system.

[0198] P2. Program instruction execution: The processor executes the mobile information to prevent misoperation confirmation interaction method according to the program instructions in the memory, including steps T-1, T-2, T-3 and T-4.

[0199] P3. Obtain audio media stream interaction from the cloud platform: The client and the cloud platform establish a broadcast connection through signaling, and finally realize audio and video interaction.

[0200] Further, the control program stored in the memory in this embodiment will be disclosed below (for reference only). This embodiment only shows the logic of its operation in the form of C++ pseudo code:

[0201] / / P1: Input data import class Device {

[0202] public:

[0203] MediaStream getMediaData();

[0204] };

[0205] Device frontendDevice;

[0206] MediaStream inputData = frontendDevice.getMediaData();

[0207] / / P2: Program instruction execution class Processor

[0208] {public:

[0209] void executeIns(T instruction); / / T represents the type of instruction};

[0210] Processor cpu;

[0211] T instruction;

[0212] cpu.executeIns(instruction);

[0213] / / P3: Get the audio and video media stream interaction of the cloud platform

[0214] class CloudPlatform

[0215] {public:

[0216] void connect(Device device);

[0217] void interact(MediaStream mediaStream);};

[0218] CloudPlatform cloudPlatform;

[0219] cloudPlatform.connect(frontendDevice);

[0220] MediaStream mediaStreamFromCloud = cloudPlatform.interact(inputData);

[0221] The above code focuses on the process of obtaining encoded and decoded data from the input device; the processor completes operations and a series of steps according to program instructions; the device establishes a broadcast connection with the cloud platform, and finally realizes the process of audio and video interaction.

[0222] Embodiment 9

[0223] On the basis of the eighth embodiment, a mobile information misoperation prevention confirmation interaction system is disclosed; the system also includes: a confirmation interaction operation smoothness evaluation module, which is used to integrate all local interaction monitoring information during the monitoring misoperation prevention confirmation interaction period and perform a smoothness evaluation on the overall operation state of the confirmation interaction implementation, and obtain smoothness evaluation data corresponding to all confirmation interaction targets; including:

[0224] When evaluating the fluency of the overall operation status of the confirmation interaction implementation, all local interaction monitoring information within the period of preventing misoperation confirmation interaction is traversed. The unit of the period of preventing misoperation confirmation interaction is day, which can be 30 days. At the same time, the normal label weight, slight abnormal label weight and severe abnormal label weight corresponding to the jump normal label, jump slight abnormal label and jump severe abnormal label are obtained and marked as CZ, QZ and ZZ respectively;

[0225] Among them, the normal label weight, the mild abnormal label weight, and the severe abnormal label weight corresponding to the jump normal label, the jump mild abnormal label, and the jump severe abnormal label can be obtained based on the historical big data of confirming the interactive operation, or can be customized based on the experience of the operation and maintenance personnel;

[0226] Normal label weights, mildly abnormal label weights, and severely abnormal label weights are used to digitize and differentiate different labels, and can provide local data support for a single confirmation interaction for the overall analysis of subsequent confirmation interaction fluency;

[0227] And, count the total number of jump normal labels, jump mild abnormal labels and jump severe abnormal labels and mark them as CZ0, QZ0 and ZZ0 respectively;

[0228] Get the confirmation interaction weight corresponding to the confirmation interaction target and mark it as JQ;

[0229] Among them, the confirmation interaction weight can be obtained based on the total profit corresponding to the confirmation interaction target, or based on the total number of transactions corresponding to the confirmation interaction target. The larger the value of the confirmation interaction weight, the greater the importance of the corresponding confirmation interaction target. The confirmation interaction weight can digitize and differentiate different confirmation interaction targets, which can effectively improve the accuracy of subsequent fluency analysis and control of different confirmation interaction targets;

[0230] The numerical values ​​of the marked data are extracted and calculated in parallel, and the interaction smoothness coefficient corresponding to the confirmed interaction target is obtained through the formula calculation; the calculation formula of the interaction smoothness coefficient is:

[0231]

[0232] In the formula, is the interaction smoothness coefficient; j1 and j2 are both constant coefficients greater than zero and 2*j1=j2; j1 can be 1.306 and j2 can be 2.612;

[0233] It should be noted that the interaction smoothness coefficient is a value used to integrate the single jump analysis data of different confirmation interactions to evaluate their overall fluency. The smaller the interaction smoothness coefficient, the more normal the fluency of the corresponding confirmation interaction and the faster the response.

[0234] When evaluating the fluency of the confirmed interaction target within the monitoring basic evaluation period according to the interaction fluency coefficient, the interaction fluency coefficient is compared with the corresponding interaction fluency threshold; the interaction fluency threshold is obtained according to the historical interaction big data of the confirmed interaction target;

[0235] If the interaction fluency coefficient is less than the interaction fluency threshold, the fluency of the confirmed interaction target is determined to be normal and a normal interaction fluency signal is generated. According to the normal interaction fluency signal, the corresponding confirmed interaction target is marked as a type of fluency target;

[0236] If the interaction fluency coefficient is not less than the interaction fluency threshold and the total number of occurrences is less than N, the fluency of the confirmed interaction target is determined to be slightly abnormal and an interaction fluency slightly abnormal signal is generated. According to the interaction fluency slightly abnormal signal, the corresponding confirmed interaction target is marked as a second-class fluency target; N is a positive integer;

[0237] If the interaction fluency coefficient is not less than the interaction fluency threshold and the total number of occurrences is not less than N, the fluency of the confirmed interaction target is determined to be severely abnormal and an interaction fluency severe abnormality signal is generated. According to the interaction fluency severe abnormality signal, the corresponding confirmed interaction target is marked as three types of fluency targets;

[0238] The interaction fluency coefficient and the corresponding interaction fluency normal signal and first-class fluency target, interaction fluency slightly abnormal signal and second-class fluency target, or interaction fluency severely abnormal signal and third-class fluency target constitute the fluency evaluation data and are uploaded to the cloud platform.

[0239] In an embodiment of the present invention, the interaction fluency coefficient is obtained by integrating the single jump analysis data of different confirmation interactions. The overall fluency of the corresponding confirmation interaction target can be analyzed and classified according to the interaction fluency coefficient, which can effectively improve the supervision effect of the confirmation interaction operation and provide data support for the subsequent quality management of the confirmation interaction operation in terms of operation fluency.

[0240] Embodiment 10

[0241] Based on the eighth embodiment, a mobile information misoperation prevention confirmation interactive system is disclosed; the system also includes:

[0242] The confirmation interaction operation abnormality monitoring module is used to monitor and count the data of abnormal feedback of the confirmation interaction target and abnormal discovery of maintenance within the monitoring basic evaluation period to obtain abnormal monitoring statistical data; including:

[0243] Statistics are collected to confirm the abnormality type reported each time and the abnormality type found each time during maintenance corresponding to the interaction target; different abnormality types are set to correspond to different abnormality weights, and the obtained abnormality type reported each time and the abnormality type found each time during maintenance are compared with all the abnormality types pre-stored in the database to obtain the corresponding abnormality weights;

[0244] Among them, feedback refers to abnormal feedback submitted by users, and maintenance discovery refers to the abnormalities found by operation and maintenance personnel in the existing operation and maintenance plan. The types of abnormalities include but are not limited to insensitive touch, no touch response, black screen, network abnormality, etc.

[0245] The anomaly weight is used to digitally represent different types of reported anomalies and maintenance-discovered anomalies, which can effectively improve the overall effect of anomaly monitoring statistics in different aspects and subsequent data analysis;

[0246] Count the total number of feedbacks corresponding to the reported abnormality type and the total number of maintenance findings corresponding to the maintenance-discovered abnormality type;

[0247] The abnormal weight corresponding to each reported abnormality type and the total number of reported abnormalities, and the abnormal weight corresponding to each maintenance discovered abnormality type and the total number of maintenance discovered abnormalities constitute abnormal monitoring statistics and are uploaded to the cloud platform.

[0248] In an embodiment of the present invention, by monitoring and counting the operational anomalies of the confirmed interactive target from different dimensions, data support of different dimensions can be provided for the subsequent operational stability quality analysis of the confirmed interactive target, which can effectively improve the accuracy of the operational stability quality analysis of the confirmed interactive target.

[0249] The confirmation interaction operation abnormality assessment module is used to process the abnormal monitoring statistical data within the monitoring basic assessment period, and to calculate and analyze the processed data to determine the operation stability state corresponding to the confirmation interaction target, and obtain the hidden danger assessment data corresponding to the confirmation interaction target; including:

[0250] When processing the abnormal monitoring statistical data, the abnormal weight and total number of abnormalities reported corresponding to each abnormality type reported in the abnormal monitoring statistical data, as well as the abnormal weight and total number of abnormalities found during maintenance corresponding to each abnormality type found during maintenance are obtained;

[0251] The anomaly weight corresponding to each feedback anomaly type is marked as BQ, and the total number of feedbacks is marked as BZ; the anomaly weight corresponding to each maintenance reverse anomaly type is marked as WQ, and the total number of maintenance findings is marked as WZ;

[0252] The values ​​of the marked data are extracted and calculated in parallel, and the interactive hidden danger coefficient corresponding to the confirmed interactive target is obtained through the formula calculation; the calculation formula of the interactive hidden danger coefficient is:

[0253]

[0254] In the formula, is the interaction hidden danger coefficient, y1 and y2 are both constant coefficients greater than zero and y1<y2, y1 can be 0.813, and y2 can be 1.625;

[0255] It should be noted that the interaction hidden danger coefficient is a value used to integrate abnormal data of different aspects of the confirmed interaction target to evaluate its overall operational stability; the larger the interaction hidden danger coefficient, the worse the operational stability of the corresponding confirmed interaction target;

[0256] When evaluating the operation stability state of the corresponding confirmed interactive target according to the interactive hidden danger coefficient, the interactive hidden danger coefficient is compared with the corresponding interactive hidden danger range;

[0257] If the interaction hidden danger coefficient is less than the minimum value of the interaction hidden danger range, it is determined that the operation stable state corresponding to the confirmed interaction target is in a normal state and a first hidden danger signal is generated, and the corresponding confirmed interaction target is marked as a type-one hidden danger target according to the first hidden danger signal;

[0258] If the interaction hidden danger coefficient is not less than the minimum value of the interaction hidden danger range and not greater than the maximum value of the interaction hidden danger range, it is determined that the operation stability state corresponding to the confirmed interaction target is in a slightly abnormal state and a second hidden danger signal is generated. According to the second hidden danger signal, the corresponding confirmed interaction target is marked as a second-class hidden danger target;

[0259] If the interaction hidden danger coefficient is greater than the maximum value of the interaction hidden danger range, it is determined that the operation stability state corresponding to the confirmed interaction target is in a severe abnormal state and a third hidden danger signal is generated. According to the third hidden danger signal, the corresponding confirmed interaction target is marked as a third-category hidden danger target;

[0260] The interactive hidden danger coefficient and the corresponding first hidden danger signal and first type hidden danger target, second hidden danger signal and second type hidden danger target, or third hidden danger signal and third type hidden danger target constitute the hidden danger assessment data corresponding to the confirmed interactive target and are uploaded to the cloud platform.

[0261] In the embodiment of the present invention, by statistically analyzing the feedback aspects and maintenance active discovery aspects of abnormalities in confirmed interaction targets and implementing data calculation and analysis classification, it is possible to intuitively and efficiently obtain the operating stability states corresponding to different confirmed interaction targets, and provide reliable data support for the dynamic management and control of subsequent maintenance of abnormal hidden dangers of confirmed interaction targets.

[0262] All the above embodiments only express the implementation methods of the relevant practical applications of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A mobile information confirmation interaction method to prevent misoperation, characterized in that: The method comprises the following steps T-1, T-2, T-3 and T-4: T-1, signaling registration: When the client accesses the platform for the first time, it will continue to send REGISTER messages until the server responds with "200 OK" to indicate successful registration. If it fails, it will try to register again after a random delay. At the same time, the client will refresh the registration periodically; T-2, data transmission interaction: After the signaling registration is successful, the server will send the directory information to the client, the client will build the content and send it to the server, the server will verify it and return the INVITE response; T-3, obtaining client audio and video codec data; wherein, extracting associated information of the audio and video codec data, the associated information includes: S3.1, Video encoding: The data collected by the camera is encoded into h264 through MediaCodec, and then packaged into PS using the RTP protocol for transmission, including: S3.11, initialize and obtain the original data returned by the device Camera2 through PreviewFrame real-time monitoring detection camera collection; S3.12, convert the collected NV21 data into h264 encoding into YUV420P data; S3.

13. Create a MediaCodec instance to set encoding format parameters and perform h264 hard encoding on YUV data; S3.14, obtaining the encoded h264 data, and encapsulating the h264 data in PS format of the national standard GB / T28181 protocol; S3.15, sending the encapsulated data to the server based on RTP / UDP; S3.2, Audio Coding: AudioRecord is used to collect sound for G711 encoding and RTP packaging transmission and transmission, including: S3.21, initialize AudioRecord to set encoding format parameters and collect audio data; S3.22, after starting recording, write the acquired audio data into the created byte buffer through audioRecord.read; S3.23, read the data in the buffer area and perform short type conversion, and encode the converted short data into G711; S3.24, the encoded data is packaged and sent to the server via the RTP protocol; S3.3, mixed stream encoding: Mix the audio and video streams through MediaMuxer, initialize MediaExtractor to extract video and audio track information from video files and audio files and obtain their formats, including: S3.31, initialize MediaExtractor to extract video and audio track information from video files and audio files; S3.32, use Media Extractor.read SampleData to read one frame of data from the video file and audio file respectively into the buffer area; S3.33, instantiate the MediaMuxer object to merge the audio and video data in the buffer and output them; S3.34, set the buffer information, including the timestamp and flag of the current frame, to determine whether it is a key frame; S3.35, using Datagram Socket for sending based on RTP / UDP; The T-4, obtains the audio media stream interaction of the cloud platform: the client and the cloud platform establish a broadcast connection through signaling, and finally realizes audio and video interaction.

2. The mobile information error-prevention confirmation interaction method according to claim 1, characterized in that: In the T-1, the signaling registration specifically includes: 1) The client initiates a registration request to the SIP server. After receiving the request, the server verifies the registration information and sends a response to the client after the verification passes. The client's authentication system and parameters are given in the header WWW-Authenticate field of the response. 2) After receiving the authentication system and parameter data, the client resends a registration request to the SIP server, providing a letter of trust and carrying authentication information in the Authorization field of the request; 3) The SIP server receives the request and verifies it again. After verification, if the client identity is legal, it sends a successful response "200 OK" to the client. If the client identity is illegal, it sends a service denial response message.

3. The mobile information error-prevention confirmation interaction method according to claim 1, characterized in that: In the T-2, the data transmission interaction: the specific steps include: S2.

1. After the signaling registration is successful, the server will send the CATALOG directory information to the client; S2.2, the client receives the CATALOG message and constructs the content, and sends it to the server after completion; S2.

3. After receiving the CATALOG information, the server verifies the content. After the verification passes, the server sends an INVITE response to the client. S2.

4. After receiving the INVITE response, the client encapsulates the INVITE information and SDP information and responds to the server. The client obtains the RTP transmission port according to the INVITE information and SDP information sent by the server, wherein the SDP information includes TCP, UDP mode and SSRC information. S2.

5. After all connections are established, the client packages the collected audio and video data into a PS stream that complies with the national standard GB / T28181 protocol and sends it to the server in the form of RTP packets; For the PS stream mode, a data stream packaging format commonly used in the national standard GB / T28181 protocol, the field describing the stream structure in the video attribute field set in the SDP file is written as: a=rtpmap:96MP2P / 90000, a=rtpmap is a fixed format, 96 is the payload type, MP2P is the encoding type, and 90000 is the clock frequency; following the previous line a=rtpmap:96MP2P / 90000, a=fmtp:96 is designed, where a=fmtp is a fixed format and 96 is the payload type; S2.

6. If the client needs to disconnect the Invite session, it will need to send a BYE request to the server. The server will disconnect after returning "200 OK"; S2.

7. If the client initiates a logout: it sends a Register command message to the server, and the Expire field in the message is set to 0, which means logout, and the session is terminated at this time.

4. The mobile information error-prevention confirmation interaction method according to claim 1, characterized in that: In the T-4, the audio media stream interaction of the cloud platform is obtained: S4.

1. The cloud platform sends a MESSAGE voice broadcast command to the client; S4.2, the client sends a MESSAGE message in response to the voice broadcast command; S4.

3. After receiving the MESSAGE response "200 OK", the client sends an INVITE message: S4.4, the client receives the INVITE and the "200 OK" response; S4.

5. After the client replies with an ACK message, it starts to read the RTP data packet, parse the RTP data packet, run the encoding logic to decode the audio data, and output it to the client for playback, thus completing the interaction.

5. The mobile information error-prevention confirmation interaction method according to claim 1, characterized in that: The client includes an intelligent head-mounted computer, a microphone, a control camera, a display optical machine and an intelligent glasses processor.

6. The mobile information error-prevention confirmation interaction method according to claim 3, characterized in that: In said T-2, said data transmission interaction further includes: a data source, a device access terminal and an international platform terminal; The data source includes data before external encoding, Android camera, screen data, encoded data H264, HEVC, RTSP or RTMP stream data, and the data type before external encoding is any one of YV12, NV21, NV12, 1420, RGB24, RGBA32, GB565; The device access terminal is used for national standard GB / T 28181 device access signaling registration, streaming media transmission, and voice broadcast reception; The international platform is the national standard GB / T 28181 platform; The data source is connected to the device access terminal, and the device access terminal is interactively connected to the international platform terminal.

7. A mobile information confirmation interactive system for preventing misoperation, characterized in that: The system comprises The confirmation interaction operation smoothness evaluation module is used to integrate all local interaction monitoring information during the monitoring period of the confirmation interaction to prevent misoperation and to evaluate the smoothness of the overall operation status of the confirmation interaction implementation, so as to obtain the smoothness evaluation data corresponding to all confirmation interaction targets; The confirmation interaction operation abnormality monitoring module is used to monitor and count the data of abnormal feedback of the confirmation interaction target and abnormal discovery of maintenance within the monitoring basic evaluation period to obtain abnormal monitoring statistical data; The confirmation interaction operation abnormality assessment module is used to process the abnormal monitoring statistical data within the monitoring basic assessment period, and to calculate and analyze the processed data to determine the operation stability state corresponding to the confirmation interaction target, and obtain the hidden danger assessment data corresponding to the confirmation interaction target; A processor and a memory coupled to the processor, wherein the memory stores program instructions, and when the program instructions are executed by the processor, the processor executes the mobile information misoperation prevention confirmation interaction method as described in any one of claims 1-6.

Images