Method for realizing remote real-time video preview control and analysis based on mobile acquisition terminal
By integrating the national standard protocol GB28181 and the private protocol WebSocket on the mobile acquisition side, remote video preview and intelligent analysis of old mobile phones and tablets as intelligent monitoring systems is realized, solving the flexibility and fixation problems of traditional video surveillance systems and improving the monitoring platform Application value and user experience.
Patent Information
- Application Number
- CN202411385298.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-12-06
AI Technical Summary
Traditional video surveillance systems rely on fixed monitoring terminals and limited network connections, limiting the flexibility and timeliness of users to obtain information. Professional surveillance cameras cannot change the monitoring scenario at will, resulting in fixed and unable to effectively utilize the old portable mobile terminal. High pixel camera and high-definition video processing capabilities.
By integrating the national standard protocol GB28181 and the private protocol WebSocket on the mobile acquisition side, it realizes interconnection with the remote platform, performs audio and video data encoding processing and intelligent analysis, supports video preview, video storage, intelligent analysis and intercom interaction, and uses old mobile phones and Tablets and other devices serve as intelligent monitoring systems.
Improve the application value and flexibility of the mobile acquisition end in the traditional monitoring platform, provide users with a convenient, efficient and personalized monitoring experience, give new value to old electronic products, reduce monitoring costs, and support rich interactive capabilities and intelligence. Alarm subscription.
Smart Images

Figure CN119094709A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of digital mobile imaging technology, and in particular to a method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal. Background Art
[0002] In today's digital and information age, the demand for real-time and remote monitoring of various scenarios is increasing day by day. Traditional monitoring systems often rely on fixed monitoring terminals, limited network connections and fixed monitoring scenarios, which limits the flexibility and timeliness of users' access to information. As for the mainstream video monitoring needs on the market, users often need to spend a lot of money to purchase camera terminals, and need to periodically subscribe to video monitoring capabilities to achieve video browsing, storage, intelligent analysis and information push.
[0003] With the rapid development of mobile Internet technology, mobile clients such as smart phones and tablet computers are becoming more and more popular. In particular, the higher the pixel of the mobile camera, the higher the corresponding video clarity, which is often higher than that of professional cameras. Its powerful processing capability and convenient network connection provide strong support for remote real-time preview, storage, and analysis. At the same time, the shortcomings of professional surveillance cameras are more obvious. The monitoring scene cannot be changed at will, and it is fixed in nature, so it needs to be purchased.
[0004] In recent years, electronic products have been rapidly updated, and there are a large number of obsolete portable mobile terminals (mobile phones, tablets, laptops, etc.) in individual households. With the widespread application of 4G, 5G, and WIFI networks, high-bandwidth and low-latency network environments provide guarantees for high-speed and stable data transmission. Therefore, it is possible to turn old mobile phones into smart monitoring systems and old laptops into security cameras. With the high-pixel cameras, high-definition video processing capabilities, analysis capabilities, and storage resources of old portable mobile terminals, they can be interconnected with traditional video surveillance platforms to turn waste into treasure and re-explore the value of these old portable mobile terminals.
[0005] Therefore, how to realize interconnection and interaction between mobile acquisition terminals and traditional video surveillance platforms, remote video preview, local storage and call of recorded videos, and intelligent data analysis and subscription has become a research hotspot. Summary of the invention
[0006] The purpose of the present invention is to provide a method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal, which greatly improves the application value and flexibility of the mobile acquisition terminal in the traditional monitoring platform, brings users a more convenient, efficient and personalized monitoring experience, and gives new value to old electronic products.
[0007] The technical solution adopted by the present invention is: The method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal comprises the following steps: Step 1: Integrate the national standard protocol and the private protocol on the mobile acquisition terminal. The mobile acquisition terminal is registered to the corresponding remote platform through the national standard protocol and the private protocol respectively. The remote platform includes the national standard monitoring platform and the non-national standard application platform; Furthermore, the national standard protocol is the GB28181 national standard protocol, and the private protocol is the WebSocket private protocol.
[0008] Step 2: The mobile acquisition terminal acquires audio and video data, and the video processor encodes the audio and video data into a specified audio and video format, and then integrates the encoded video stream to transmit it to the video storage and the intelligent analyzer respectively; Specifically, the mobile acquisition end starts to load the audio and video codec initialization parameters, including camera parameters, MIC parameters, whether the screen is blocked, the screen frame, and adding watermarks, and starts the camera and MIC to obtain real-time audio and video data. The audio and video data are encoded into video H264 and audio PCMA data respectively through the video encoder and audio decoder.
[0009] Furthermore, after the audio and video data is encoded and processed into the specified audio and video format, the audio and video data queue waits for the video data processor to consume it. If there is no video preview request, intelligent analysis configuration, or storage configuration, the video data processor directly discards the data. After the video data processor processes the video masking, watermarking, framing, and compression, if the platform has initiated a video preview request, the encoded video stream is pushed to the national standard stream encoder and diverted to the video storage for subsequent processing.
[0010] Step 3: The video storage device converts the encoded video stream data into a file of a specified format (MP4 file) according to the storage configuration requirements and stores it in the storage space of the mobile acquisition terminal. At the same time, the intelligent analyzer extracts key frames from the video stream for image analysis and processing, and determines in real time whether there is an abnormal sound in the audio data. When an alarm situation is detected, an alarm is broadcast and pushed to the remote platform simultaneously. Step 4: The remote platform initiates a remote video preview request to the mobile acquisition end through the corresponding protocol (GB28181, WebSocket protocol), and carries the platform streaming information; Step 5: The mobile acquisition terminal determines whether the current state can be used for video streaming; if so, the mobile acquisition terminal starts to initialize the streaming device information and start the network detection service and then executes step 6; otherwise, it feedbacks to the remote platform that the current streaming conditions are not met and executes step 5; Step 6: The mobile acquisition end determines whether the remote video preview request is a real-time preview request; if so, obtains the currently encoded video stream from the video processor; otherwise, extracts the encoded video stream corresponding to the time of the remote video preview request from the video storage; Step 7: The mobile acquisition end pushes the extracted encoded video stream to the protocol encapsulator to encapsulate it into audio and video data that conforms to the UDP / TCP protocol, and then the stream pusher pushes it to the streaming information address provided by the remote platform, and the user end accesses the streaming information address to realize remote video preview; Step 8: The network detection service detects whether the current network quality is lower than the set mobile acquisition end network traffic threshold; if so, the mobile phone camera is triggered to reduce the video source data bit rate and start the video processor to start the video compression technology; otherwise, the camera bit rate of the mobile acquisition end is maintained or restored, and the video compression function of the video processor is turned off; Step 9: The user sends an intercom interaction request to the mobile acquisition end through the remote platform according to the needs. The mobile acquisition end determines whether the intercom interaction is possible in the current state. If so, the mobile acquisition end initializes the stream receiver (service IP, port, protocol type), starts to monitor the audio data pushed by the platform and puts it into the audio stream queue to wait for consumption before executing step 10. Otherwise, it feedbacks to the remote platform that the conditions for intercom interaction are not met and executes step 11. Step 10: The audio decoder consumes the audio data in the audio stream queue, performs ps decoding, pes decoding, sampling rate conversion, noise reduction and other processing on the audio data, and then pushes it to the mobile phone Mic for playback.
[0011] Furthermore, during the intercom interaction process of step 10, the remote platform conducts instant text interaction with the mobile acquisition terminal through GB28181 and WebSocket protocols.
[0012] Step 11, the mobile acquisition end determines whether a request to end the preview is received; if so, the streaming is stopped and ends; otherwise, step 7 is executed.
[0013] In addition, step 3 is to allow new video stream data to record and overwrite old video stream data if the video stream data stored in the mobile acquisition terminal exceeds the storage period.
[0014] Furthermore, the storage configuration requirements in step 3 include the length of the recording time, the size of the clip, and the storage period; Furthermore, the image analysis processing in step 3 includes people / motion detection and license plate recognition.
[0015] Specifically, in step 3, the intelligent analyzer extracts frames at a fixed frequency (2 key frame intervals), and pushes the extracted key frames to the personnel / motion detection and license plate recognition threads for processing and analysis. It also obtains audio data through real-time analysis, calculates the characteristic values of the short-time energy and zero-crossing rate of the audio, sets the characteristic value threshold, and identifies it as an abnormal sound if it exceeds the threshold. It broadcasts the detected alarm data as an audio alarm and pushes it to the platform through the GB28181 extension protocol and WebSocket protocol.
[0016] Furthermore, in step 3, when the alarm is broadcasted, it is decided whether to notify the user of the analyzed alarm information by SMS / telephone according to the requirements of the configuration information.
[0017] Furthermore, in step 5, the mobile acquisition terminal responds via GB28181 and WebSocket protocols; Furthermore, the video stream encoded in step 7 is encoded as a PS stream by a national standard stream encoder and then pushed to the protocol encapsulator.
[0018] Furthermore, PS stream encoding includes audio and video processing of key ps packet header, pes packet header, seq, ssrc, and timestamp.
[0019] Furthermore, in step 7, the user sends a preview control request to the mobile acquisition end through the remote platform; the mobile acquisition end performs video picture control processing according to the preview control request; the video picture control processing includes video picture masking switch / area, picture frame switch / table frame information (vertex xy coordinate points and length and width information), watermark switch and watermark information (vertex xy axis coordinates and watermark text).
[0020] The present invention also discloses a remote real-time video preview control and analysis system based on a mobile acquisition terminal, which includes a mobile acquisition terminal, a national standard monitoring platform, a non-national standard application platform and a user terminal. The mobile acquisition terminal is integrated with a national standard protocol and a private protocol. The mobile acquisition terminal is connected to the national standard monitoring platform through the national standard protocol, and the mobile acquisition terminal is connected to the non-national standard application platform through a private protocol. The national standard monitoring platform and the non-national standard application platform are respectively connected to the user terminal; the national standard monitoring platform and the non-national standard application platform call the end-side capabilities of video preview, video storage, intelligent analysis, and intercom interaction of the mobile acquisition terminal through GB28181 and WebSocket protocols.
[0021] Furthermore, the mobile acquisition terminal is a mobile phone, a tablet, or a laptop computer. In addition, the systems supported by the mobile acquisition terminal are Android, IOS, Hongmeng, and Window systems.
[0022] Furthermore, the mobile acquisition terminal has the following capabilities: audio and video preview function, video storage and review function, intelligent analysis and alarm function, and mobile acquisition terminal interactive function; the audio and video preview function includes video preview, video preview control, and video compression push; the video storage and review function includes remote video access; the intelligent analysis and alarm function includes abnormal sound detection, personnel detection / motion detection, license plate recognition, file broadcast, and SMS / telephone alarm function; the mobile acquisition terminal interactive function includes voice intercom function, instant text message function, and picture frame function.
[0023] The present invention adopts the above technical solution. The mobile acquisition terminal mainly adopts the national standard GB28181 protocol to achieve interconnection with the traditional national standard video surveillance platform, and is oriented to the application platform with non-national standard GB28181 capabilities, mainly using the WebSocket long connection protocol, and using the json string as the message body to provide private protocol customization. With the help of the national standard GB28181 / private customized protocol, the remote preview and control functions of the mobile acquisition terminal and the voice intercom capabilities can be smoothly realized.
[0024] Moreover, for standards not covered by the national standard protocol, the present invention also extends the protocol for the national standard GB28181, realizing support for video screen masking, front and rear camera switching, screenshots, etc. on the mobile acquisition end. At the same time, it has the function of intelligent analysis and alarm subscription, and users can customize alarm rules according to their own needs to ensure timely acquisition of key information. In addition, it also supports rich interactive capabilities such as picture frames and text interactions. For example, users can mark picture frames of specific areas on the video screen, or interact with the monitoring system through text input to put forward specific analysis requirements or instructions. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments; Figure 1 It is a schematic diagram of the principle architecture of the method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal of the present invention; Figure 2 This is a schematic diagram of the video preview process of the mobile acquisition terminal of the present invention; Figure 3 This is a schematic diagram of the video storage process of the mobile acquisition terminal of the present invention; Figure 4 This is a schematic diagram of the intelligent analysis process of the mobile acquisition terminal of the present invention; Figure 5 It is a schematic diagram of the interactive process of the mobile collection terminal of the present invention. Implementation
[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0027] like Figures 1 to 5 As shown in one, the present invention discloses a method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal, which comprises the following steps: Step 1: Integrate the national standard protocol and the private protocol on the mobile acquisition terminal. The mobile acquisition terminal is registered to the corresponding remote platform through the national standard protocol and the private protocol respectively. The remote platform includes the national standard monitoring platform and the non-national standard application platform; Furthermore, the national standard protocol is the GB28181 national standard protocol, and the private protocol is the WebSocket private protocol.
[0028] Step 2: The mobile acquisition terminal acquires audio and video data, and the video processor encodes the audio and video data into a specified audio and video format, and then integrates the encoded video stream to transmit it to the video storage and the intelligent analyzer respectively; Specifically, the mobile acquisition end starts to load the audio and video codec initialization parameters, including camera parameters, MIC parameters, whether the screen is blocked, the screen frame, and adding watermarks, and starts the camera and MIC to obtain real-time audio and video data. The audio and video data are encoded into video H264 and audio PCMA data respectively through the video encoder and audio decoder.
[0029] Furthermore, after the audio and video data is encoded and processed into the specified audio and video format, the audio and video data queue waits for the video data processor to consume it. If there is no video preview request, intelligent analysis configuration, or storage configuration, the video data processor directly discards the data. After the video data processor processes the video masking, watermarking, framing, and compression, if the platform has initiated a video preview request, the encoded video stream is pushed to the national standard stream encoder and diverted to the video storage for subsequent processing.
[0030] Step 3: The video storage device converts the encoded video stream data into a file of a specified format (MP4 file) according to the storage configuration requirements and stores it in the storage space of the mobile acquisition terminal. At the same time, the intelligent analyzer extracts key frames from the video stream for image analysis and processing, and determines in real time whether there is an abnormal sound in the audio data. When an alarm situation is detected, an alarm is broadcast and pushed to the remote platform simultaneously. Step 4: The remote platform initiates a remote video preview request to the mobile acquisition end through the corresponding protocol (GB28181, WebSocket protocol), and carries the platform streaming information; Step 5: The mobile acquisition terminal determines whether the current state can be used for video streaming; if so, the mobile acquisition terminal starts to initialize the streaming device information and start the network detection service and then executes step 6; otherwise, it feedbacks to the remote platform that the current streaming conditions are not met and executes step 5; Step 6: The mobile acquisition end determines whether the remote video preview request is a real-time preview request; if so, obtains the currently encoded video stream from the video processor; otherwise, extracts the encoded video stream corresponding to the time of the remote video preview request from the video storage; Step 7: The mobile acquisition end pushes the extracted encoded video stream to the protocol encapsulator to encapsulate it into audio and video data that conforms to the UDP / TCP protocol, and then the stream pusher pushes it to the streaming information address provided by the remote platform, and the user end accesses the streaming information address to realize remote video preview; Step 8: The network detection service detects whether the current network quality is lower than the set mobile acquisition end network traffic threshold; if so, the mobile phone camera is triggered to reduce the video source data bit rate and start the video processor to start the video compression technology; otherwise, the camera bit rate of the mobile acquisition end is maintained or restored, and the video compression function of the video processor is turned off; Step 9: The user sends an intercom interaction request to the mobile acquisition end through the remote platform according to the needs. The mobile acquisition end determines whether the intercom interaction is possible in the current state. If so, the mobile acquisition end initializes the stream receiver (service IP, port, protocol type), starts to monitor the audio data pushed by the platform and puts it into the audio stream queue to wait for consumption before executing step 10. Otherwise, it feedbacks to the remote platform that the conditions for intercom interaction are not met and executes step 11. Step 10: The audio decoder consumes the audio data in the audio stream queue, performs ps decoding, pes decoding, sampling rate conversion, noise reduction and other processing on the audio data, and then pushes it to the mobile phone Mic for playback.
[0031] Furthermore, during the intercom interaction process of step 10, the remote platform conducts instant text interaction with the mobile acquisition terminal through GB28181 and WebSocket protocols.
[0032] Step 11, the mobile acquisition end determines whether a request to end the preview is received; if so, the streaming is stopped and ends; otherwise, step 7 is executed.
[0033] In addition, step 3 is to allow new video stream data to record and overwrite old video stream data if the video stream data stored in the mobile acquisition terminal exceeds the storage period.
[0034] Furthermore, the storage configuration requirements in step 3 include the length of the recording time, the size of the clip, and the storage period; Furthermore, the image analysis processing in step 3 includes people / motion detection and license plate recognition.
[0035] Specifically, in step 3, the intelligent analyzer extracts frames at a fixed frequency (2 key frame intervals), and pushes the extracted key frames to the personnel / motion detection and license plate recognition threads for processing and analysis. It also obtains audio data through real-time analysis, calculates the characteristic values of the short-time energy and zero-crossing rate of the audio, sets the characteristic value threshold, and identifies it as an abnormal sound if it exceeds the threshold. It broadcasts the detected alarm data as an audio alarm and pushes it to the platform through the GB28181 extension protocol and WebSocket protocol.
[0036] Furthermore, in step 3, when the alarm is broadcasted, it is decided whether to notify the user of the analyzed alarm information by SMS / telephone according to the requirements of the configuration information.
[0037] Furthermore, in step 5, the mobile acquisition terminal responds via GB28181 and WebSocket protocols; Furthermore, the video stream encoded in step 7 is encoded as a PS stream by a national standard stream encoder and then pushed to the protocol encapsulator.
[0038] Furthermore, PS stream encoding includes audio and video processing of key ps packet header, pes packet header, seq, ssrc, and timestamp.
[0039] Furthermore, in step 7, the user sends a preview control request to the mobile acquisition end through the remote platform; the mobile acquisition end performs video picture control processing according to the preview control request; the video picture control processing includes video picture masking switch / area, picture frame switch / table frame information (vertex xy coordinate points and length and width information), watermark switch and watermark information (vertex xy axis coordinates and watermark text).
[0040] Furthermore, the present invention also discloses a remote real-time video preview control and analysis system based on a mobile acquisition terminal, which includes a mobile acquisition terminal, a national standard monitoring platform, a non-national standard application platform and a user terminal. The mobile acquisition terminal integrates a national standard protocol and a private protocol. The mobile acquisition terminal is connected to the national standard monitoring platform through the national standard protocol, and the mobile acquisition terminal is connected to the non-national standard application platform through a private protocol. The national standard monitoring platform and the non-national standard application platform are respectively connected to the user terminal; the national standard monitoring platform and the non-national standard application platform call the end-side capabilities of the mobile acquisition terminal for video preview, video storage, intelligent analysis, and intercom interaction through GB28181 and WebSocket protocols.
[0041] Furthermore, the mobile acquisition terminal is a mobile phone, a tablet, or a laptop computer. In addition, the systems supported by the mobile acquisition terminal are Android, IOS, Hongmeng, and Window systems.
[0042] Furthermore, the mobile acquisition terminal has the following capabilities: audio and video preview function, video storage and review function, intelligent analysis and alarm function, and mobile acquisition terminal interactive function; the audio and video preview function includes video preview, video preview control, and video compression push; the video storage and review function includes remote video access; the intelligent analysis and alarm function includes abnormal sound detection, personnel detection / motion detection, license plate recognition, file broadcast, and SMS / telephone alarm function; the mobile acquisition terminal interactive function includes voice intercom function, instant text message function, and picture frame function.
[0043] Specifically, the mobile client has the following capabilities: 1. Compatibility: The model of the present invention is compatible with mobile phones, tablets, laptops, etc., and the system adaptation: Android, iOS, Hongmeng, Window, etc.
[0044] 2. Basic protocol: The method of the present invention has GB28181 protocol + extended protocol and private customized protocol, so as to realize the remote monitoring access of the national standard video monitoring platform / non-national standard platform to the mobile acquisition terminal.
[0045] 3. Capabilities: ①Audio and video preview: Video preview: By obtaining the audio and video data from the mobile acquisition end camera and MIC, the camera YUV video raw data and the MIC PCM raw audio data are encoded into audio and video data that conforms to the national standard stream and pushed to the platform.
[0046] Video preview control: By receiving instructions from the video surveillance platform, the real-time video preview is configured with screen masking, screen watermarking, and front and rear camera switching. After receiving the instructions, the mobile client encodes the video stream in real time and pushes it to the video surveillance platform.
[0047] Video compression push: Real-time detection of the network status of the mobile acquisition end. In the case of weak network or network fluctuations, the push video bit rate is reduced or the video is compressed for video streaming.
[0048] ② Video storage and review: Remote video access: Encode real-time video + audio data into MP4 files and save them to the local storage of the mobile acquisition terminal. You can configure storage policies according to file size, video clip time, and storage cycle. Use streaming technology to implement GB28181 protocol remote access and push them to the specified server IP and port to achieve remote preview effects. You can use commands to achieve double-speed streaming and double-speed playback effects. ③ Intelligent analysis and alarm: Abnormal sound detection: The audio data is obtained through real-time analysis of the mic, and the short-time energy, zero-crossing rate and other characteristics of the audio are calculated. The characteristic value threshold is set. If it exceeds the threshold, it is identified as an abnormal sound, and the detection alarm is pushed to the platform.
[0049] Personnel detection / motion detection: By acquiring camera video data in real time, with the help of Android automatic face detection capability (FaceDeDetection) + Opencv face detection capability, personnel detection / motion detection is realized, and the detection alarm is pushed to the platform.
[0050] License plate recognition: By acquiring camera video data in real time, video frames are extracted at a specific frequency, and the extracted video is analyzed in real time using the Tesseract OCR open source library. The analysis results are pushed to the platform through the GB28181 extended protocol.
[0051] File broadcast, SMS / phone alarm: Through local intelligent analysis of alarms, configure file mobile phone broadcasts, and call system SMS and phone capabilities to push detection alarms. The platform can be remotely configured for this capability.
[0052] ④Interaction on mobile acquisition terminal: Voice intercom capability: Traditional monitoring platforms can initiate voice intercom capabilities through the GB28181 protocol, and the mobile client will establish UDP / TCP audio data monitoring and receiving encoding for playback.
[0053] Instant text messaging: Enables two-way instant text interaction capabilities through the protocol.
[0054] Picture frame: By receiving the picture frame information sent by the platform, the video picture frame of the mobile acquisition end can be displayed.
[0055] The specific principle of the present invention is described in detail below: like Figure 1 As shown in the framework interaction diagram, the interaction between modules is described as follows: 1. The mobile acquisition terminal integrates GB28181 and WebSocket communication protocols; 2. The mobile acquisition terminal maintains a connection with the platform through GB28181 and WebSocket protocols, and performs registration and heartbeat packet activity status 3. The national standard platform / non-national standard platform uses GB28181 and WebSocket protocols to call the mobile client's preview, recording, intelligence, interaction and other end-side capabilities; like Figure 2 As shown in the figure, the video preview process on the mobile acquisition end is described as follows: 1. The mobile acquisition terminal registers to the platform through GB28181 and WebSocket protocols. The mobile acquisition terminal starts to load the audio and video codec initialization parameters, including camera parameters, MIC parameters, whether the screen is blocked, the screen frame, and adding watermarks. It also starts the camera and MIC to obtain the audio and video data. 2. After the audio and video data are encoded into video H264 and audio PCMA data respectively by the video encoder and audio decoder, the audio and video data queue waits for the video data processor to consume. If there is no video preview control, intelligent analysis configuration, and storage configuration, the video data processor will directly discard the data. 3. After the video data processor performs video masking, watermarking, framing, and compression, if the platform has initiated video preview control, the encoded video stream will be pushed to the national standard stream encoder and diverted to the video storage for subsequent processing.
[0056] 4. The platform initiates a remote video preview request to the mobile acquisition terminal through GB28181 and WebSocket protocols, and carries the platform streaming information (IP, PORT, protocol (UDP / TCP), SSRC (synchronization source identifier, distinguishing video sources)); 5. The mobile acquisition terminal determines whether the current state can be used for video streaming, and responds through GB28181 and WebSocket protocols; 6. If the conditions for streaming are met, the mobile acquisition end starts to initialize the streaming device information and start the network detection service.
[0057] 7. Start the national standard stream encoder to encode the PS stream, including the key PS header, PES header, seq, ssrc, timestamp and other audio and video processing, and then push it to the protocol encapsulator to encapsulate it into audio and video data that conforms to the UDP / TCP protocol 8. After the protocol stream is encapsulated, it is pushed by the stream pusher to the streaming information address provided by the platform. 9. The platform can control the video screen mask switch / area, picture frame switch / frame information (vertex xy coordinate points and length and width information), watermark switch and watermark information (vertex xy axis coordinates and watermark text) of the mobile acquisition terminal through GB28181 and WebSocket protocols. 10. Network detection detects the quality of mobile network and WIFI network, sets the network traffic threshold of mobile acquisition end, if it is lower than the network traffic threshold, triggers the mobile phone camera to reduce the video source data bit rate and starts the video processor to start the video compression technology, if the network quality is higher than the network traffic threshold, restores the camera bit rate and turns off the video compression function of the video processor.
[0058] like Figure 3As shown in the figure, the video storage processing flow of the mobile acquisition end is described as follows: 1. Mobile registers to the platform through GB28181 and WebSocket protocols. The mobile acquisition end starts to load the initialization parameters of audio and video codec, including camera parameters, MIC parameters, whether the screen is blocked, the screen frame, add watermarks, start the camera, MIC, and obtain the audio and video data. Move the camera and MIC to obtain the audio and video data. 2. After the audio and video data are encoded into video H264 and audio PCMA data respectively by the video encoder and audio decoder, the audio and video data queue waits for the video data processor to consume. If there is no video preview control, intelligent analysis configuration or storage configuration, the video data processor will directly discard the data.
[0059] 3. After the video data processor processes the video masking, watermarking, framing, and compression, if the platform has initiated video preview control, the encoded video stream will be pushed to the national standard stream encoder and diverted to the video storage and intelligent analyzer for subsequent processing; 4. The video storage device converts the audio and video data into MP4 files according to the storage configuration requirements (video recording time, clip size, storage period) and stores them in the mobile phone storage space. If the storage period is exceeded, the video will be overwritten. 5. The platform returns the relative storage list information when requesting the local video file list information through GB28181 and WebSocket protocol; 6. The platform requests the mobile acquisition end to preview the video clip file through GB28181 and WebSocket protocols. After the mobile acquisition end streams the MP4 file through the video streamer, it is pushed to the designated server according to the server IP and port given by the platform.
[0060] like Figure 4 As shown in the figure, the intelligent analysis and processing flow of the mobile acquisition end is described as follows: 1. Mobile registers to the platform through GB28181 and WebSocket protocols. The mobile acquisition end starts to load the initialization parameters of audio and video codec, including camera parameters, MIC parameters, whether the screen is blocked, the screen frame, and adding watermarks. It also starts the camera and MIC to obtain the audio and video data. 2. After the audio and video data are encoded into video H264 and audio PCMA data respectively by the video encoder and audio decoder, the audio and video data queue waits for the video data processor to consume. If there is no video preview control, intelligent analysis configuration or storage configuration, the video data processor will directly discard the data.
[0061] 3. After the video data processor processes the video masking, watermarking, framing, and compression, if the platform has initiated video preview control, the encoded video stream will be pushed to the national standard stream encoder and diverted to the video storage and intelligent analyzer for subsequent processing; 4. The intelligent analyzer extracts frames at a fixed frequency (2 key frame intervals), and pushes the extracted key frames to the personnel / motion detection and license plate recognition threads for processing and analysis. It obtains audio data through real-time analysis, calculates the characteristic values of the audio's short-time energy and zero-crossing rate, sets the characteristic value threshold, and identifies it as an abnormal sound if it exceeds the threshold. It broadcasts the detected alarm data as an audio alarm and pushes it to the platform through the GB28181 extension protocol and WebSocket protocol.
[0062] 5. Determine whether to notify the user of the analyzed alarm information via SMS / phone call based on the configuration information.
[0063] like Figure 5 As shown in the figure, the video interaction processing flow of the mobile acquisition end is described as follows: 1. Mobile registers to the platform through GB28181 and WebSocket protocols.
[0064] 2. The platform uses GB28181 and WebSocket protocols to invite the mobile acquisition end to interact and carry the platform streaming address information.
[0065] 3. After receiving the intercom interaction request, the mobile terminal detects the network environment of the mobile acquisition terminal and whether there is an intercom link, and then responds to the platform.
[0066] 4. After the mobile acquisition terminal has the conditions for audio intercom interaction, it initializes the stream receiver (service IP, port, protocol type), starts to monitor the audio data pushed by the platform and puts it into the audio stream queue waiting for consumption 5. The audio decoder consumes the audio data in the audio stream queue, performs ps decoding, pes decoding, sampling rate conversion, noise reduction and other processing on the audio data, and then pushes it to the mobile phone Mic for playback.
[0067] 6. The platform uses GB28181 and WebSocket protocols to conduct instant text interaction on the mobile collection end.
[0068] The mobile acquisition terminal of the present invention mainly adopts the national standard GB28181 protocol to achieve interconnection with the traditional national standard video monitoring platform, and is oriented to the application platform of non-national standard GB28181 capability, with WebSocket long connection protocol as the main, json string as the message body, and provides private protocol customization. With the help of national standard GB28181 / private customized protocol, the remote preview and control functions of the mobile acquisition terminal and the ability of voice intercom can be smoothly realized. Not only that, for standards not covered by the national standard protocol, the present invention also extends the protocol for national standard GB28181, and realizes support for video screen masking, front and rear camera switching, screenshots, etc. of the mobile acquisition terminal. At the same time, it has the function of intelligent analysis and alarm subscription, and users can customize alarm rules according to their own needs to ensure timely acquisition of key information. In addition, it also supports rich interactive capabilities such as picture frames and text interactions. For example, users can mark picture frames of specific areas on the video screen, or interact with the monitoring system through text input to put forward specific analysis requirements or instructions. Through the above series of functional expansion and innovation, the application value and flexibility of mobile acquisition terminals in traditional monitoring platforms have been greatly improved, bringing users a more convenient, efficient and personalized monitoring experience, and giving new value to old electronic products.
[0069] The present invention adopts the above technical scheme to realize mobile video monitoring through old electronic products, and realizes local video storage. Users do not need to purchase specific surveillance camera products for scene monitoring, which reduces the user's monitoring cost. By providing national standard protocols and extended protocols, and private customized protocols, the mobile acquisition terminal can be remotely monitored and accessed in traditional national standard video monitoring platforms and non-national standard platforms, greatly expanding the application scope and flexibility of the mobile acquisition terminal in the monitoring field. The present invention enhances the user experience: it has rich functions, such as audio and video preview control, video storage and review, intelligent analysis alarms and mobile acquisition terminal interaction, etc., which brings users a convenient, efficient and personalized monitoring experience. The present invention ensures the acquisition of key information: it supports intelligent analysis alarm subscription, users can customize alarm rules, and obtain key information in a timely manner. At the same time, there are multiple alarm push methods to ensure the timeliness and effectiveness of information transmission.
[0070] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians of the art without making creative work are within the scope of protection of the present application.
Claims
1. A method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal, characterized in that: It includes the following steps: Step 1: Integrate the national standard protocol and the private protocol on the mobile acquisition terminal. The mobile acquisition terminal is registered to the corresponding remote platform through the national standard protocol and the private protocol respectively. The remote platform includes the national standard monitoring platform and the non-national standard application platform; Step 2: The mobile acquisition terminal acquires audio and video data, and the video processor encodes the audio and video data into a specified audio and video format, and then integrates the encoded video stream to transmit it to the video storage and the intelligent analyzer respectively; Step 3: The video storage device converts the encoded video stream data into a file of a specified format according to the storage configuration requirements and stores it in the storage space of the mobile acquisition terminal. At the same time, the intelligent analyzer extracts key frames from the video stream for image analysis and processing, and determines in real time whether there is an abnormal sound in the audio data. When an alarm situation is detected, an alarm is broadcast and pushed to the remote platform simultaneously. Step 4: The remote platform initiates a remote video preview request to the mobile acquisition end through the corresponding protocol, and carries the platform streaming information; Step 5: The mobile acquisition terminal determines whether the current state can be used for video streaming; if so, the mobile acquisition terminal starts to initialize the streaming device information and start the network detection service and then executes step 6; otherwise, it feedbacks to the remote platform that the current streaming conditions are not met and executes step 5; Step 6: The mobile acquisition end determines whether the remote video preview request is a real-time preview request; If yes, get the currently encoded video stream from the video processor; Otherwise, extract the encoded video stream corresponding to the time of the remote video preview request from the video storage; Step 7: The mobile acquisition end pushes the extracted encoded video stream to the protocol encapsulator to encapsulate it into audio and video data that conforms to the UDP / TCP protocol, and then the stream pusher pushes it to the streaming information address provided by the remote platform, and the user end accesses the streaming information address to realize remote video preview; Step 8: The network detection service detects whether the current network quality is lower than the set mobile acquisition end network traffic threshold; if so, the mobile phone camera is triggered to reduce the video source data bit rate and start the video processor to start the video compression technology; otherwise, the camera bit rate of the mobile acquisition end is maintained or restored, and the video compression function of the video processor is turned off; Step 9: The user sends an intercom interaction request to the mobile acquisition end through the remote platform according to the demand, and the mobile acquisition end determines whether the intercom interaction is possible in the current state; if so, the mobile acquisition end initializes the stream receiver, starts to listen to the audio data pushed by the platform and puts it into the audio stream queue waiting for consumption before executing step 10; otherwise, it feedbacks to the remote platform that the conditions for intercom interaction are not met and executes step 11; Step 10: The audio decoder consumes the audio data in the audio stream queue, performs ps decoding, pes decoding, sampling rate conversion, and noise reduction on the audio data, and then pushes it to the mobile phone Mic for playback; Step 11: The mobile acquisition end determines whether a request to end the preview is received; if yes, the streaming is stopped and the process ends; Otherwise, go to step 7.
2. The method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal according to claim 1, characterized in that: The national standard protocol is GB28181 national standard protocol, and the private protocol is the WebSocket private protocol.
3. The method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal according to claim 1, characterized in that: In step 2, the mobile acquisition end starts to load the audio and video codec initialization parameters, including camera parameters, MIC parameters, whether the screen is masked, the screen frame, and watermarking, and starts the camera and MIC to obtain real-time audio and video data. The camera and MIC are activated to obtain real-time audio and video data; the audio and video data are encoded into video H264 and audio PCMA data respectively through the video encoder and audio decoder; after the audio and video data are encoded and processed into the specified audio and video format, the audio and video data queue waits for the video data processor to consume. When there is no video preview request, intelligent analysis configuration, and storage configuration, the video data processor directly discards the data; after the video data processor performs video masking, watermarking, framing, and compression, when the platform has initiated a video preview request, the encoded video stream is pushed to the national standard stream encoder and diverted to the video storage for subsequent processing.
4. The method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal according to claim 1, characterized in that: The storage configuration requirements in step 3 include the length of recording time, the size of the clip, and the storage period; the image analysis processing includes personnel / motion detection and license plate recognition; when the video stream data stored in the mobile acquisition terminal exceeds the storage period, the new video stream data is allowed to record and overwrite the old video stream data.
5. The method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal according to claim 1, characterized in that: In step 3, the intelligent analyzer extracts frames at a fixed frequency, pushes the extracted key frames to the personnel / motion detection and license plate recognition threads for processing and analysis, obtains audio data through real-time analysis, and calculates the characteristic values of the audio's short-time energy and zero-crossing rate; Set the characteristic value threshold, and if it exceeds the threshold, it will be identified as an abnormal sound; the detected alarm data will be broadcast as an audio alarm and pushed to the remote platform.
6. The method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal according to claim 5, characterized in that: In step 3, when the alarm is broadcast, it is decided whether to notify the user of the analyzed alarm information by SMS / phone according to the requirements of the configuration information.
7. The method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal according to claim 1, characterized in that: In step 5, the mobile acquisition end responds through the GB28181 and WebSocket protocols.
8. The method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal according to claim 1, characterized in that: The video stream encoded in step 7 is pushed to the protocol encapsulator after being PS stream encoded by the national standard stream encoder. The PS stream encoding includes audio and video processing of key ps packet header, pes packet header, seq, ssrc, and timestamp.
9. The method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal according to claim 1, characterized in that: In step 7, the user sends a preview control request to the mobile acquisition end through the remote platform; the mobile acquisition end performs video picture control processing according to the preview control request; the video picture control processing includes video picture masking switch / area, picture frame switch / table frame information, watermark switch and watermark information.
10. The method for realizing remote real-time video preview control and analysis based on a mobile acquisition terminal according to claim 1, characterized in that: During the intercom interaction process in step 10, the remote platform conducts real-time text interaction with the mobile acquisition end through GB28181 and WebSocket protocols.
