Rescue scene unmanned aerial vehicle video processing method and device
By adopting the method of working together with the drone in rescue scenarios, smooth video acquisition and real-time target recognition are achieved, solving the problems of low video processing efficiency and insufficient data integration in traditional technologies, and providing strong data support.
Patent Information
- Application Number
- CN202411425742.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In rescue scenarios, traditional drone video processing technology faces problems such as unstable network communication, inefficient video processing and lack of real-time target recognition and data integration functions.
Through the coordinated work of the server and the drone, smooth video acquisition, real-time target recognition and synchronous processing of video and subtitles are achieved. Ad hoc network architecture is adopted, and video live broadcast and recording is used using MQTT and rtmp protocols, and based on preset The object recognition algorithm model performs object recognition, and generates a subtitle file with one subtitle in one second and is inserted into the video.
It realizes smooth video acquisition and real-time target recognition, provides strong data support, and provides stable and accurate data support for decision-making analysis of rescue operations.
Smart Images

Figure CN118972626A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of video processing, and in particular, relates to a method and device for processing drone videos of rescue scenarios. Background Art
[0002] In modern rescue scenarios, such as fire rescue, disaster response and other emergency situations, drones are widely used as an efficient information collection tool to provide real-time on-site videos. These videos are crucial for the rear command department to formulate rescue plans, assess the disaster situation, and coordinate rescue operations. However, traditional drone video processing technology faces some challenges in practical applications, such as live broadcast freezes caused by unstable network communication, inefficient video processing, and lack of real-time target recognition and data integration functions. Summary of the invention
[0003] The purpose of this application is to overcome the problems existing in the above-mentioned prior art and provide a method and device for processing drone videos of rescue scenes.
[0004] This application provides a method for processing drone videos in rescue scenarios, including: Steps to start video acquisition: The server sends a live broadcast start command to the drone; The start live broadcast instruction controls the drone to start acquiring the video stream; The drone pushes the video stream to the server; The server performs live broadcast and recording of the video stream, and records the start timestamp of the video stream; Video end collection steps: The server sends a live broadcast closing instruction to the drone; The live broadcast shutdown instruction controls the drone to stop pushing the video stream; The server stores the recorded video and records the end timestamp; Video processing steps: The server sends a target identification instruction to the drone; The drone performs object recognition on the video according to the target recognition instruction and based on a preset target recognition algorithm model, and converts the recognition result into a digital message and sends it to the server; The server stores the digital message and records a timestamp; The server generates a subtitle file according to the format of one subtitle per second by using the attribute information of the drone and the digital message; The server inserts the subtitle file into the video file.
[0005] Optionally, the server sends a live broadcast start command to the drone, including: Specifies the video encoding format used by the drone.
[0006] Optionally, the server performs live broadcast and recording of the video stream, including: Real-time monitoring of video quality, including resolution, frame rate, and network latency; An adjustment instruction is sent to the drone to correct the video quality according to the resolution, frame rate and network delay.
[0007] Optionally, the server sends a live broadcast start instruction to the drone, including: sending a live broadcast start instruction to a plurality of drones; Each of the drones independently pushes the video stream to the server.
[0008] Optionally, the server remotely updates the target recognition algorithm model on the drone.
[0009] The present application also provides a rescue scene drone video processing system, including: a server, a drone; The server sends a live broadcast start instruction to the drone; the live broadcast start instruction controls the drone to start acquiring the video stream; the drone pushes the video stream to the server; the server broadcasts and records the video stream, and records the start timestamp of the video stream; The server sends a live broadcast closing instruction to the drone; the live broadcast closing instruction controls the drone to stop pushing the video stream; the server stores the recorded video and records the end timestamp; The server sends a target recognition instruction to the drone; the drone recognizes objects in the video based on a preset target recognition algorithm model according to the target recognition instruction, converts the recognition result into a digital message and sends it to the server; the server stores the digital message and records a timestamp; the server generates a subtitle file according to the format of one subtitle per second using the attribute information of the drone and the digital message; the server inserts the subtitle file into the video file.
[0010] Optionally, the server sends a live broadcast start command to the drone, including: Specifies the video encoding format used by the drone.
[0011] Optionally, the server performs live broadcast and recording of the video stream, including: Real-time monitoring of video quality, including resolution, frame rate, and network latency; An adjustment instruction is sent to the drone to correct the video quality according to the resolution, frame rate and network delay.
[0012] Optionally, the server sends a live broadcast start instruction to the drone, including: sending a live broadcast start instruction to a plurality of drones; Each of the drones independently pushes the video stream to the server.
[0013] Optionally, the server remotely updates the target recognition algorithm model on the drone.
[0014] The beneficial effects of this application are: The present application provides a method for processing unmanned aerial vehicle (UAV) videos in a rescue scenario, comprising: a video start acquisition step: a server sends a live broadcast start instruction to the UAV; the live broadcast start instruction controls the UAV to start acquiring a video stream; the UAV pushes the video stream to the server; the server broadcasts and records the video stream, and records the start timestamp of the video stream; a video end acquisition step: the server sends a live broadcast stop instruction to the UAV; the live broadcast stop instruction controls the UAV to stop pushing the video stream; the server stores the recorded video and records the end timestamp; a video processing step: the server sends a target recognition instruction to the UAV; the UAV performs object recognition on the video based on a preset target recognition algorithm model according to the target recognition instruction, converts the recognition result into a digital message and sends it to the server; the server stores the digital message and records the timestamp; the server generates a subtitle file according to the attribute information of the UAV and the digital message in the format of one subtitle per second; the server inserts the subtitle file into the video file. This application achieves smooth video acquisition, real-time target recognition, and synchronous processing of video and subtitles through the collaborative work of the server and the drone, providing strong data support for decision-making analysis of rescue operations. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a schematic diagram of the architecture of the self-organizing network device in this application; Figure 2 It is a schematic diagram of the video processing business process in this application; Figure 3 This is a schematic diagram of the process of adding subtitles to videos in this application; Figure 4 This is a schematic diagram of the video collection process in this application; Figure 5 This is a schematic diagram of the video end acquisition process in this application; Figure 6 It is a schematic diagram of the video processing flow in this application. DETAILED DESCRIPTION
[0016] The present application is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present application and implement it.
[0017] The present application provides a method for processing drone videos of rescue scenarios. To implement the method, the present application provides a self-organizing network.
[0018] like Figure 1 As shown, the ad hoc network includes five parts: an ad hoc network 1, a server device 2, a terminal device 3, a server service 4, and a terminal service 5.
[0019] The equipment of the ad hoc network 1 includes: an ad hoc network radio 1-1, a switch 1-2, and a router 1-3. After being successfully built, they serve as basic components for network communication and provide various network connection forms such as wireless wifi and wired network.
[0020] The self-organizing server is deployed as follows: The server application system is deployed to the server. The server device 2 selection includes but is not limited to mobile device 2-1, laptop computer 2-2, server 2-3, PC computer 2-4, and can be implemented according to different scenarios or existing hardware device selection.
[0021] Install and deploy the srs streaming media server and emqx IoT message server on the server, which can interact with the drone 3-1 on the department side.
[0022] In this application, the system framework selected is SpringBoot to build a service based on the mqtt+webRtc protocol.
[0023] The end device 3 is preferably a drone 3-1, which interacts with the server through the mqtt protocol and the rtmp protocol.
[0024] The functions of the server-side service 4 mainly include other server-side services 4-1, video streaming 4-2, and heartbeat detection services 4-3, wherein the video streaming 4-2 and the heartbeat detection services 4-3 are related to the functional points of the present invention. The server pulls the live stream pushed by the drone 3-1 and regularly detects the heartbeat information of the drone 3-1.
[0025] The functions of the end service 5 mainly include other client services 5-1, video streaming 5-2, and heartbeat information sending 5-3, among which video streaming 5-2 and heartbeat information sending 5-3 are related to the functional points of the present invention, and mainly complete the pushing of video live stream to the server and the sending of heartbeat information.
[0026] like Figure 2As shown, the process of drone 3-1 includes: drone 3-1 goes online, initiates MQTT connection a-1, establishes connection a-3, early warning information a-2, accepts instructions a-4, RTMP push stream a-5, receives instructions a-6, targets identify a-7, accepts instructions a-8, stops pushing stream a-9.
[0027] The server process includes: server startup, listening to mqtt b-1, establishing connection b-2, storing drone attributes b-3, starting live broadcast b-4, webRtc streaming b-5, starting video recording b-6, video live broadcast b-7, starting target recognition b-8, storing recognition information b-9, closing live broadcast b-10, generating video b-11, storing minio b-12, ending task b-13, srt subtitle file b-14, inserting subtitles into video b-15.
[0028] Based on the above-mentioned self-organizing network, the steps of the scene-aiding drone 3-1 video processing method provided in this application include: video acquisition, video acquisition end and video processing steps.
[0029] Steps to start video acquisition: S101, the server sends a live broadcast start command to drone 3-1; S102, the live broadcast start instruction controls the drone 3-1 to start acquiring the video stream; S103, the drone 3-1 pushes the video stream to the server; S104: The server broadcasts and records the video stream, and records a start timestamp of the video stream.
[0030] Specifically, based on the self-organizing network described in this application, the emqx IoT message server is deployed and started using the mobile device 2-1. The server is responsible for receiving and forwarding MQTT messages, and serves as a bridge for communication between the drone 3-1 and the server. The srs streaming media server is deployed and started using the laptop 2-2. The server supports push and pull operations of video streams and is the core component for processing the video stream of the drone 3-1. The minio distributed storage service is deployed and started using the server 2-3. The service is used to store video files shot by the drone 3-1 for a long time to ensure the security and accessibility of the data.
[0031] After the drone 3-1 is started and online, it will initiate a connection request to the server. If the drone 3-1 fails to connect to the server, a warning message will be sent to relevant personnel through the router 1-3 so that the problem can be promptly checked.
[0032] After the connection is successful, UAV 3-1 establishes a stable connection with the server and maintains the connection status through the heartbeat mechanism to ensure the real-time and stability of communication.
[0033] The server controls the drone 3-1 to start live broadcast: The server sends an mqtt command to start live streaming to drone 3-1 through PC 2-4. After receiving the command, drone 3-1 prepares to start recording the video stream. Drone 3-1 starts recording the video stream according to the received command.
[0034] Drone 3-1 pushes the recorded video stream to the SRS streaming server on the server side through protocols such as RTMP.
[0035] The server pulls the video stream pushed by drone 3-1 from the srs streaming server through the webRtc protocol and starts live video broadcast. At the same time, the server starts recording the video stream and records the start timestamp of the video stream for subsequent management and playback.
[0036] The server broadcasts the pulled video stream live through the webRtc protocol or other streaming protocols, and users can watch the images taken by drone 3-1 in real time through clients such as web pages and mobile applications.
[0037] While the server is live streaming, it records the video stream to the minio distributed storage service to ensure persistent storage of video data. During the recording process, the server will record the start timestamp of the video for subsequent management and analysis of the video.
[0038] The server is responsible for storing the attribute information sent by UAV 3-1 at a fixed frequency, such as speed, altitude, longitude and latitude, etc. This information is of great significance for subsequent data analysis and UAV 3-1 management.
[0039] During the entire process, if the connection is interrupted, data transmission is abnormal, etc., the system will promptly notify relevant personnel through the corresponding early warning mechanism for processing.
[0040] Video end collection steps: S201, the server sends a live broadcast closing instruction to the drone; When it is necessary to end the live video broadcast, the server will send a command to close the live broadcast to the drone 3-1. This command is sent through a pre-set communication protocol (such as the MQTT protocol) to ensure that the command can be accurately transmitted to the drone 3-1.
[0041] For example: Use the server to generate and send an MQTT command to close the live broadcast.
[0042] S202, the live broadcast shutdown instruction controls the drone 3-1 to stop pushing the video stream; After receiving the live broadcast shutdown command sent by the server, drone 3-1 will immediately stop pushing the video stream to the server. The processing module inside drone 3-1 will parse the received MQTT command and execute the operation of stopping the push stream according to the command content.
[0043] For example: Use drone 3-1 to receive and parse the MQTT command sent by the server. After drone 3-1 parses the command to close the live broadcast, stop the RTMP (Real-time Messaging Protocol) live streaming.
[0044] S203: The server stores the recorded video and records the end timestamp.
[0045] After confirming that drone 3-1 has stopped pushing the video stream, the server will store the previously recorded video file and record the end timestamp of the video for subsequent management and retrieval.
[0046] The server will check whether the real-time video stream has stopped. Once it is confirmed to have stopped, the video saving mechanism will be triggered to save the video file to the specified storage location and record the video storage path, file name, and timestamp of the end of recording.
[0047] For example, when the server fails to receive the real-time video stream, the video saving mechanism is triggered, and the recorded video file is automatically saved, while recording key information such as the video storage path, file name, and end recording timestamp.
[0048] Video processing steps: S301, the server sends a target recognition command to the drone 3-1; On the server side, when it is necessary to identify a specific object in the video transmitted by the drone 3-1, the server side will send a clear target recognition instruction to the drone 3-1. This instruction contains information such as the type of object to be identified and the recognition conditions, so that the drone 3-1 can perform the corresponding recognition task accordingly.
[0049] The server sends the target identification command to the drone 3-1 through its command sending module, and the drone 3-1 receives and parses the command through its command receiving module.
[0050] S302, the drone 3-1 performs object recognition on the video according to the target recognition instruction and based on a preset target recognition algorithm model, and converts the recognition result into a digital message and sends it to the server; After receiving the target recognition command, UAV 3-1 will use its internal preset target recognition algorithm model to identify objects in the real-time video stream according to the object type and recognition conditions specified in the command. This algorithm model is built based on deep learning, image processing and other technologies, and can accurately and quickly identify specific objects in the video.
[0051] The drone 3-1 captures the video stream through its video processing module and sends it to the target recognition algorithm module for processing. The target recognition algorithm module recognizes the objects in the video stream according to the preset algorithm model.
[0052] After successfully identifying the object in the video, the drone 3-1 will convert the identification result into a specific digital message format. This format usually contains information such as the type, location, size, etc. of the identified object, so that the server can easily store and analyze it. Then, the drone 3-1 sends this digital message to the server.
[0053] The drone 3-1 converts the recognition result into a digital message through its data conversion module, and then sends the digital message to the server through its communication module.
[0054] S303, the server stores the digital message and records a timestamp; After receiving the digital message sent by drone 3-1, the server will store it. At the same time, the server will also record the timestamp of receiving the digital message so that the time point of the recognition result can be accurately known during the subsequent re-analysis.
[0055] The server saves the digital message into a database or a file system through its storage module, and records the timestamp of the received digital message through its timestamp recording module.
[0056] S304: The server generates a subtitle file according to the format of one subtitle per second based on the attribute information of the drone 3-1 and the digital message.
[0057] like Figure 3 As shown, the server module includes: obtaining subtitle content A, generating srt subtitle file B, inserting srt file into video C and video storage minio D.
[0058] Get subtitle content A: Get the content to generate the SRT subtitle file. This module collects or receives the specific information that needs to be converted into the SRT subtitle file, which is crucial for the subsequent generation of subtitles.
[0059] The information can come from a variety of sources such as sensor data, GPS tracking systems, people counting systems, etc.
[0060] The data fields include: Flight speed: record the speed information of drone 3-1 when performing a mission.
[0061] Flight altitude: record its flight altitude.
[0062] Latitude and longitude: mark the current location.
[0063] Timestamp: records the specific time of data collection, which is used for the subsequent synchronous display of subtitles.
[0064] Number of people: refers to the number of people or other important personnel statistics at the rescue site.
[0065] Organize the collected data into a format suitable for generating captions, including converting numbers into understandable text descriptions.
[0066] Generate srt subtitle file B: Generate srt subtitle file according to the acquired subtitle content.
[0067] Convert the information collected by module A into the standard format of SRT subtitle file.
[0068] Write the corresponding subtitle text according to the flight speed, altitude, longitude and latitude information, for example, "Current flight speed: XX km / h, altitude: XX meters, position: XX latitude, XX longitude".
[0069] Assign appropriate timestamps to each subtitle entry to ensure synchronization between the subtitles and the video content. The timestamps must follow the "hour: minute: second, millisecond" standard of the SRT format and set the start and end time of each subtitle.
[0070] Write the file content according to the SRT file format (the first line is the index, the second line is the timestamp, the third line is the subtitle content, and the fourth line is a blank line) and save it as a .srt file.
[0071] S305: The server inserts the subtitle file into the video file.
[0072] Insert srt file into video C: Use ffmpeg technology to insert srt file into video file Merge the generated SRT subtitle file with the video file to achieve superimposed display of subtitles.
[0073] Use the ffmpeg command line tool to write a command that includes the input video file, SRT subtitle file, and output video file path.
[0074] Run the command, ffmpeg will read the video file and subtitle file, and overlay the subtitles onto the video to generate a new video file with subtitles.
[0075] Check the command execution results to ensure that the subtitles are successfully superimposed and handle any errors that occur.
[0076] Video storage minio D: Store video files with subtitles in the high-performance distributed storage service minio and update the database The processed video files are securely stored in the distributed storage system and the database records are updated for subsequent retrieval and use.
[0077] Use the API or client tool provided by minio to upload the video file with subtitles to the specified bucket.
[0078] Record the detailed information of the newly uploaded video in the database, including the file name, storage location, upload time, etc., for subsequent retrieval and management.
[0079] Log all operations during upload and database update for easy tracing and debugging.
[0080] The embodiments and / or implementation methods described above are only used to illustrate the preferred embodiments and / or implementation methods for realizing the technology of the present invention, and are not intended to limit the implementation methods of the technology of the present invention in any form. Any technical personnel in this field may make slight changes or modifications to other equivalent embodiments without departing from the scope of the technical means disclosed in the content of the present invention, but they should still be regarded as technologies or embodiments that are essentially the same as the present invention.
[0081] The present application also provides a rescue scene drone video processing system, including: a server, a drone; The server sends a live broadcast start instruction to the drone; the live broadcast start instruction controls the drone to start recording the video stream; the drone pushes the video stream to the server; the server broadcasts and records the video stream, and records the start timestamp of the video stream; The server sends a live broadcast closing instruction to the drone; the live broadcast closing instruction controls the drone to stop pushing the video stream; the server stores the recorded video and records the end timestamp; The server sends a target recognition instruction to the drone; the drone recognizes objects in the video based on a preset target recognition algorithm model according to the target recognition instruction, converts the recognition result into a digital message and sends it to the server; the server stores the digital message and records a timestamp; the server generates a subtitle file according to the format of one subtitle per second using the attribute information of the drone and the digital message; the server inserts the subtitle file into the video file.
[0082] Furthermore, the server sends a live broadcast start command to the drone, including: Specifies the video encoding format used by the drone.
[0083] Furthermore, the server performs live broadcast and recording of the video stream, including: Real-time monitoring of video quality, including resolution, frame rate, and network latency; An adjustment instruction is sent to the drone to correct the video quality according to the resolution, frame rate and network delay.
[0084] Further, the server sends a live broadcast start instruction to the drone, including: sending a live broadcast start instruction to a plurality of drones; Each of the drones independently pushes the video stream to the server.
[0085] Furthermore, the server remotely updates the target recognition algorithm model on the drone.
[0086] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and its core ideas of this application. The above is only the preferred implementation method of this application. It should be pointed out that due to the limitations of textual expression and the objective existence of infinite specific structures, ordinary technicians in this technical field can make some improvements, modifications or changes without departing from the principles of this application, and combine the above technical features in an appropriate manner; the improvements, modifications, changes or combinations, or the direct application of the concept and technical solution of the invention to other occasions without improvement, should be regarded as the scope of protection of this application.
Claims
1. A method for processing drone videos of rescue scenes, characterized in that: include: Steps to start video acquisition: The server sends a live broadcast start command to the drone; The start live broadcast instruction controls the drone to start acquiring the video stream; The drone pushes the video stream to the server; The server performs live broadcast and recording of the video stream, and records the start timestamp of the video stream; Video end collection steps: The server sends a live broadcast closing instruction to the drone; The live broadcast shutdown instruction controls the drone to stop pushing the video stream; The server stores the recorded video and records the end timestamp; Video processing steps: The server sends a target identification instruction to the drone; The drone performs object recognition on the video according to the target recognition instruction and based on a preset target recognition algorithm model, and converts the recognition result into a digital message and sends it to the server; The server stores the digital message and records a timestamp; The server generates a subtitle file according to the format of one subtitle per second by using the attribute information of the drone and the digital message; The server inserts the subtitle file into the video file.
2. A method for processing rescue scene drone videos according to claim 1, characterized in that: The server sends a live broadcast start command to the drone, including: Specifies the video encoding format used by the drone.
3. The method for processing a rescue scene drone video according to claim 1, characterized in that: The server performs live broadcast and recording of the video stream, including: Real-time monitoring of video quality, including resolution, frame rate, and network latency; An adjustment instruction is sent to the drone to correct the video quality according to the resolution, frame rate and network delay.
4. The method for processing rescue scene drone video according to claim 1, characterized in that: The server sends a live broadcast start instruction to the drone, including: sending the live broadcast start instruction to multiple drones; Each of the drones independently pushes the video stream to the server.
5. The method for processing rescue scene drone video according to claim 1, characterized in that: The server remotely updates the target recognition algorithm model on the drone.
6. A rescue scene drone video processing system, characterized in that: include: Server, drone; The server sends a live broadcast start instruction to the drone; The start live broadcast instruction controls the drone to start acquiring the video stream; the drone pushes the video stream to the server; The server performs live broadcast and recording of the video stream, and records the start timestamp of the video stream; The server sends a live broadcast closing instruction to the drone; the live broadcast closing instruction controls the drone to stop pushing the video stream; The server stores the recorded video and records the end timestamp; The server sends a target recognition instruction to the drone; the drone performs object recognition on the video based on the target recognition instruction and a preset target recognition algorithm model, converts the recognition result into a digital message and sends it to the server; the server stores the digital message and records a timestamp; The server generates a subtitle file according to the format of one subtitle per second by using the attribute information of the drone and the digital message; The server inserts the subtitle file into the video file.
7. A rescue scene drone video processing system according to claim 6, characterized in that: The server sends a live broadcast start command to the drone, including: Specifies the video encoding format used by the drone.
8. The rescue scene drone video processing system according to claim 6, characterized in that: The server performs live broadcast and recording of the video stream, including: Real-time monitoring of video quality, including resolution, frame rate, and network latency; An adjustment instruction is sent to the drone to correct the video quality according to the resolution, frame rate and network delay.
9. A rescue scene drone video processing system according to claim 6, characterized in that: The server sends a live broadcast start instruction to the drone, including: sending the live broadcast start instruction to multiple drones; Each of the drones independently pushes the video stream to the server.
10. A rescue scene drone video processing system according to claim 6, characterized in that: The server remotely updates the target recognition algorithm model on the drone.
Citation Information
Patent Citations
Method and device for displaying electronic nameplate in video conference
CN112307800A
Platform guiding system and method
CN112464859A
Specific target object intelligent discrimination method based on urban low-altitude remote sensing data
CN113612966A
Emergency communication platform and communication method of unmanned aerial vehicle and ground station in emergency scene
CN116449861A
Target positioning method and device, electronic equipment and storage medium
CN116912716A
