Robot multi-sensor information fusion video display method, device, system and storage medium

Abstract

The application discloses a kind of robot multi-sensor information fusion video display method, device, system and storage medium, the system includes the robot with dynamic camera and multiple sensors, server and client browser, server is equipped with fusion video streaming media platform, fusion video streaming media platform receives the video stream and audio stream and other data sent by robot, after decoding and transcoding etc., using uniform format package, platform receives the application initiated from the browser of customer, and according to application, corresponding audio and video stream and other sensor data are issued, for customer to consult.The application can be according to the viewing application submitted by client browser, and the multiple real-time sensor data is fused after video stream data, and is issued to customer to consult, so that customer can obtain the relevant information expressed in multiple forms simultaneously through the same display interface, more intuitively perceive the relationship between data variation and video live.

Description

Technical Field

[0001] This invention relates to the field of audio and video processing technology, specifically to a method, apparatus, system, and storage medium for displaying video by fusing multi-sensor information from a robot. Background Technology

[0002] Currently, the common video display solution for industrial / security inspection robots is a layout where a large amount of environmental inspection data is the main focus on the page, supplemented by real-time monitoring video. The data and video are displayed separately without integration. When using traditional cameras, this layout can display as much data as possible, and customers only need to understand it in conjunction with the real-time video stream, which does not carry additional algorithmic information.

[0003] Modern industrial inspection robots increasingly employ algorithms and artificial intelligence, incorporating more and more information into video streams, such as human / object recognition bounding boxes and annotations. This makes the traditional information display increasingly fragmented and cluttered, hindering customers' ability to intuitively understand the relationship between the environmental inspection data collected by the robot and the real-time video algorithm detection information. Furthermore, the robots lack sufficient sensitivity to changes in key data such as noise waveforms, temperature, humidity, and location information, preventing customers from clearly perceiving the relationship between data fluctuations and the live video feed.

[0004] Furthermore, common industrial camera video streams are in RTSP format and are viewed through the manufacturer's dedicated client. However, there are compatibility issues when customers view these streams in modern HTML5 browsers, preventing direct playback. In such cases, customers need to install a plugin provided by the manufacturer in their browser to view the video directly, or developers need to integrate the manufacturer's complex, non-open-source SDK into the webpage, which obviously causes significant inconvenience. Summary of the Invention

[0005] Technical Objective: To address the aforementioned technical problems, this invention proposes a method, device, system, and storage medium for displaying multi-sensor information fusion video of a robot. It integrates and displays multiple real-time sensor data with video stream data, enabling customers to simultaneously obtain multiple key information through the same display interface of a conventional browser, achieving the goal of integrating and displaying multiple data and video streams at a lower cost.

[0006] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution:

[0007] A method for displaying video by fusing information from multiple sensors in a robot, characterized by its application in a client browser, includes the following steps:

[0008] Send a viewing request to the converged video streaming platform hosted on the server;

[0009] Receive video streams and sensor data in FLV format, corresponding to the viewing request, transmitted from the converged video streaming platform;

[0010] The received video stream and sensor data stream are decoded using flv.js, and the decoded audio stream is sent to the Web Audio API interface.

[0011] The Web Audio API interface analyzes and processes the decoded audio stream to obtain audio sampling information;

[0012] The pre-created second canvas is used to receive and hide the decoded video stream, and the pre-created third canvas is used to receive the audio sampling information processed by the Web Audio API interface, and to draw and hide the corresponding soundprint.

[0013] The real-time images of the second and third canvases are copied to the pre-created first canvas frame by frame, pixel by pixel. The first, second, and third canvases are pre-created using the canvas tag in HTML5.

[0014] The client browser displays real-time information from the first canvas.

[0015] Preferably, the client browser pre-creates a canvas using the canvas tag in HTML5, which also includes a hidden fourth canvas. The fourth canvas is used to receive infrared video streams, perform keying on target people or objects in the infrared video streams, and hide the keyed images. The real-time image of the fourth canvas is copied onto the first canvas.

[0016] Preferably, the viewing request includes the robot ID, camera ID, and sensor ID.

[0017] Preferably, the converged video streaming platform performs the following steps:

[0018] Receive the video stream sent by the robot, use FFMPEG software decoding to convert the video stream from RTSP stream to RTMP stream, encapsulate the converted video stream in FLV format and store it in the database;

[0019] Receive sensor data sent by the robot, including audio streams, use FFMPGE software decoding on the audio streams, convert the audio streams from any format to AAC format audio streams, encapsulate the transcoded audio streams and other sensor data besides the audio streams using Flv format and store them in the database.

[0020] It receives viewing requests from client browsers, retrieves corresponding video streams and sensor data encapsulated in FLV format from the database based on the requests, and sends them to the client browser, which then creates and displays the merged video stream.

[0021] A robot multi-sensor information fusion video display device, characterized in that it is applied to a client browser and includes:

[0022] The customer request sending module is used to create a viewing request and transmit the viewing request to the integrated video streaming platform mounted on the server.

[0023] The data receiving module is used to receive video streams and sensor data encapsulated in FLV format transmitted by the converged video streaming platform;

[0024] The decoding module uses flv.js to decode the video stream and sensor data stream transmitted by the converged video streaming platform, and the decoded audio stream is sent to the Web Audio API interface.

[0025] The Web Audio API interface module is used to analyze and process the decoded audio stream to obtain audio sampling information;

[0026] The canvas creation module is used to create canvases using the canvas tag in HTML5.

[0027] The display module receives and hides the decoded video stream using a pre-created second canvas, receives audio sampling information processed by the Web Audio API using a pre-created third canvas, draws and hides the corresponding soundprint, and copies the real-time images of the second and third canvases to the pre-created first canvas frame by frame, and displays the real-time information on the first canvas.

[0028] Preferably, the display module further includes a hidden display fourth canvas, used to receive infrared video streams, perform keying on target people or objects in the infrared video streams, and display the keyed images in a hidden manner. The real-time image of the fourth canvas is copied onto the first canvas.

[0029] Preferably, the converged video streaming platform includes:

[0030] The video stream processing module is used to receive the video stream sent by the robot, and to use FFMPEG software decoding to convert the video stream from RTSP stream to RTMP stream. The transcoded video stream is then encapsulated in FLV format and stored in the database.

[0031] The audio stream receiving and processing module is used to receive sensor data sent by the robot, including audio streams, and to use FFMPGE software decoding to convert the audio streams from any format to AAC format audio streams. The transcoded audio streams and other sensor data besides the audio streams are encapsulated in Flv format and stored in the database.

[0032] The customer request processing module is used to receive viewing requests initiated by the client browser, and to retrieve the corresponding video stream and sensor data encapsulated in FLV format from the database based on the viewing request, and send them to the client browser, which then creates and displays the fused video stream.

[0033] A robot multi-sensor information fusion video display system, characterized in that it includes a robot equipped with a dynamic camera and multiple sensors, a server, and one or more client browsers;

[0034] The sensors are used to detect information about the robot's environment, including one or more of the following: microphone, sound card, pickup, temperature sensor, humidity sensor, positioning device, and harmful gas concentration sensor.

[0035] The client browser is equipped with the robot's multi-sensor information fusion video display device.

[0036] A computer-readable storage medium is characterized in that it is used to store a computer program; wherein the computer program implements the method when executed by a processor.

[0037] Beneficial effects: Due to the adoption of the above technical solution, the present invention has the following beneficial effects:

[0038] The method and system of this invention, by mounting a converged video streaming platform on a server, communicates with the client browser and robot through the platform front-end and platform back-end respectively. Based on the actual viewing request submitted by the client browser, multiple real-time sensor data and video stream data are fused and displayed. This allows customers to obtain multiple related information in different forms simultaneously through a single display interface in a regular browser. This overcomes the problems existing in the prior art and achieves the purpose of fusion display of multiple data and video streams at a lower cost. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the system structure of the information fusion video streaming media display scheme in Embodiment 3;

[0040] Figure 2 This is a schematic diagram illustrating the integrated video streaming platform in Example 3. Detailed Implementation

[0041] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0042] Example 1

[0043] This embodiment proposes a method for displaying video by fusing information from multiple sensors in a robot, applied to a client browser, including the following steps:

[0044] Send a viewing request to the converged video streaming platform hosted on the server;

[0045] Receive video streams and sensor data in FLV format, corresponding to the viewing request, transmitted from the converged video streaming platform;

[0046] The received video stream and sensor data stream are decoded using flv.js, and the decoded audio stream is sent to the Web Audio API interface.

[0047] The Web Audio API interface analyzes and processes the decoded audio stream to obtain audio sampling information;

[0048] The pre-created second canvas is used to receive and hide the decoded video stream, and the pre-created third canvas is used to receive the audio sampling information processed by the Web Audio API interface, and to draw and hide the corresponding soundprint.

[0049] The real-time images of the second and third canvases are copied to the pre-created first canvas frame by frame, pixel by pixel. The first, second, and third canvases are pre-created using the canvas tag in HTML5.

[0050] The client browser displays real-time information from the first canvas.

[0051] The client browser pre-creates a canvas using the canvas tag in HTML5, which also includes a hidden and displayed fourth canvas. The fourth canvas is used to receive infrared video streams, perform keying on the target person or object in the infrared video stream, and hide the keyed image. The real-time image of the fourth canvas is copied onto the first canvas.

[0052] The converged video streaming platform performs the following steps:

[0053] Receive the video stream sent by the robot, use FFMPEG software decoding to convert the video stream from RTSP stream to RTMP stream, encapsulate the converted video stream in FLV format and store it in the database;

[0054] Receive sensor data sent by the robot, including audio streams, use FFMPGE software decoding on the audio streams, convert the audio streams from any format to AAC format audio streams, encapsulate the transcoded audio streams and other sensor data besides the audio streams using Flv format and store them in the database.

[0055] It receives viewing requests from client browsers, retrieves corresponding video streams and sensor data encapsulated in FLV format from the database based on the requests, and sends them to the client browser, which then creates and displays the merged video stream.

[0056] Example 2

[0057] This embodiment proposes a robot multi-sensor information fusion video display device, applied to a client browser, including:

[0058] The customer request sending module is used to create a viewing request and transmit the viewing request to the integrated video streaming platform mounted on the server.

[0059] The data receiving module is used to receive video streams and sensor data encapsulated in FLV format transmitted by the converged video streaming platform;

[0060] The decoding module uses flv.js to decode the video stream and sensor data stream transmitted by the converged video streaming platform, and the decoded audio stream is sent to the Web Audio API interface.

[0061] The Web Audio API interface module is used to analyze and process the decoded audio stream to obtain audio sampling information;

[0062] The canvas creation module is used to create canvases using the canvas tag in HTML5.

[0063] The display module receives and hides the decoded video stream using a pre-created second canvas, receives audio sampling information processed by the Web Audio API using a pre-created third canvas, draws and hides the corresponding soundprint, and copies the real-time images of the second and third canvases to the pre-created first canvas frame by frame, and displays the real-time information on the first canvas.

[0064] The display module also includes a hidden fourth canvas, which is used to receive infrared video streams, perform keying on target people or objects in the infrared video streams, and hide the keyed images. The real-time image of the fourth canvas is copied onto the first canvas.

[0065] Among them, the converged video streaming platform includes:

[0066] The video stream processing module is used to receive the video stream sent by the robot, and to use FFMPEG software decoding to convert the video stream from RTSP stream to RTMP stream. The transcoded video stream is then encapsulated in FLV format and stored in the database.

[0067] The audio stream receiving and processing module is used to receive sensor data sent by the robot, including audio streams, and to use FFMPGE software decoding to convert the audio streams from any format to AAC format audio streams. The transcoded audio streams and other sensor data besides the audio streams are encapsulated in Flv format and stored in the database.

[0068] The customer request processing module is used to receive viewing requests initiated by the client browser, and to retrieve the corresponding video stream and sensor data encapsulated in FLV format from the database based on the viewing request, and send them to the client browser, which then creates and displays the fused video stream.

[0069] Example 3

[0070] This embodiment proposes a robot multi-sensor information fusion video display system, including a robot equipped with a dynamic camera and multiple sensors, a server, and one or more client browsers; the sensors are used to detect environmental information of the robot, including one or more of the following: microphone, sound card, pickup, temperature sensor, humidity sensor, positioning device, and harmful gas concentration sensor; the client browser is equipped with the robot multi-sensor information fusion video display device.

[0071] by Figure 1 The system shown, for example, includes components mounted on a robot:

[0072] Dynamic cameras, including visible light and infrared cameras, can be loaded with AI algorithm chips to achieve edge recognition. The robot reads the camera video stream via the RTSP protocol and transmits it to the server video stream processing module.

[0073] Multiple sensors are supported, including microphones / sound cards / pickups, temperature sensors, humidity sensors, positioning devices, and harmful gas concentration sensors. The robot reads the environmental information obtained by the sensors through hardware protocols and uploads it to the server database for storage.

[0074] The system also includes a server that communicates with the robot and a client that communicates with the server.

[0075] This embodiment enhances the richness of video stream information by equipping the robot with multiple sensors and using AI recognition algorithms, and integrates various data with the video stream into the display platform, allowing customers to focus more on key information.

[0076] The server is equipped with a converged video streaming platform. The platform's backend receives various data streams reported by each inspection robot and processes them accordingly, including database storage and retrieval, video stream transcoding, and audio stream transcoding. The platform's frontend receives client viewing requests, converges and displays video streams, audio streams, and other sensor data. The robots are responsible for collecting data from their mounted hardware devices and data generated by related software during runtime, and reporting this data to the server's relevant services using various protocols. Regarding other sensor data, including gas sensors, temperature sensors, and photoelectric sensors, this data is generally in digital format and is read from the MySQL database without requiring dedicated transcoding; it is uniformly displayed digitally on the frontend. The platform's frontend is the client browser, and the converged video stream display is executed within the client browser. The server platform receives each video stream through the platform's backend. If the server does not receive an active request from the client (i.e., the corresponding webpage is not opened on the platform's frontend), it will not display the current robot's data. Upon receiving an active request from the client, the server executes the relevant processes.

[0077] The client sends a viewing request, including the robot ID and camera ID. The platform determines the information to be queried, and the methods include: the product manager presets basic query logic based on the client's business needs, and the platform then queries according to the preset query logic. Alternatively, the query can be based on logic set by the client. For example, to query data of an inspection robot on a production line: 1) When only visible light is being viewed, the robot ID and visible light camera ID are sent. The platform returns data including visible light video stream, audio stream, and sensor data. Then, the video stream is drawn on the client's browser, the audio data is analyzed, and other sensor data is displayed; 2) When viewing both visible light and infrared light, the video streams are merged and displayed. In this case, the robot ID, the visible light and infrared light IDs are sent. The platform returns data including visible light video stream, infrared video stream, audio stream, and sensor data. Then, the infrared video is cut out according to preset rules on the client's browser and merged with the visible light video stream for display, while simultaneously analyzing audio data and displaying other sensor data; the visible light camera ID is also sent, and the platform returns data including visible light video stream, audio stream, and sensor data.

[0078] Regarding receiving viewing requests and distributing information, the platform uses Nginx as a relay station to receive requests initiated by customers from their browsers and distribute the corresponding audio and video streams and other sensor data according to the requests for customers to view.

[0079] The main video module in the platform display solution of this information fusion video streaming media display solution is as follows: Figure 2As shown, by utilizing the HTML5 canvas technology of modern browsers and the Web Audio API technology, the dynamic display of information such as temperature, humidity, location, and sound collected by sensors is achieved by fusing them with real-time video streams.

[0080] The client's browser uses flv.js to decode the streaming media and streams the video stream to the native HTML5 video tag for playback, while the audio stream is sent to the Web Audio API interface for analysis and processing.

[0081] Regarding video stream transcoding, FFMPEG software decoding is used to transcode the video stream from RTSP to RTMP and encapsulate it in FLV format;

[0082] Regarding audio stream transcoding, FFMPGE software decoding is used to convert the audio stream from any format to an AAC format audio stream, and then it is encapsulated in FLV format.

[0083] In this embodiment, as Figure 2 As shown, there are three canvases, all created using the HTML5 canvas tag. The size of the first canvas is roughly the same as the combined size of the second and third canvases (this is an example for ease of illustration). The first canvas is explicitly displayed, while the second and third canvases are hidden. The video tag is hidden and is only used to receive and play the video stream decoded by flv.js. The second canvas copies the video stream from the video tag frame by frame, pixel by pixel, and continuously refreshes the canvas. The third canvas receives audio sampling information processed by the Web Audio API and performs corresponding voiceprint rendering. Finally, the real-time images from the second and third canvases are copied onto the first canvas, achieving a merged display effect, so the client only sees the content of the first canvas.

[0084] Canvas supports keying. In real-time video streams, if needed, an additional video tag can be added to receive the infrared video stream. A fourth canvas can then be added to key out the target person / object from the infrared camera and place it above the first canvas, achieving a fusion display of visible and infrared light. Given limited server AI computing power, this is a low-cost solution for fusing and displaying AI recognition results from dual visible and infrared video streams. This embodiment reduces the development costs incurred by developers using vendor-specific, non-open-source SDKs.

[0085] Furthermore, the full names and Chinese meanings of the abbreviations involved in this invention are as follows:

[0086] RTSP: Real Time Streaming Protocol.

[0087] SDK: Software Development Kit.

[0088] HTML5: HyperText Markup Language 5.

[0089] FFMPEG: Fast Forward MPEG, a video file format that uses fast compression.

[0090] AAC: Advanced Audio Coding.

[0091] Nginx: engine x, a high-performance HTTP and reverse proxy web server.

[0092] In another embodiment of the present invention, a computer-readable storage medium is provided for storing a computer program; wherein the computer program, when executed by a processor, implements the method described in Embodiment 1 or Embodiment 2.

[0093] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for displaying video by fusing information from multiple sensors in a robot, characterized in that, Applied to the client browser, the steps include: Send a viewing request to the converged video streaming platform hosted on the server; Receive video stream and sensor data in FLV format corresponding to the viewing request from the converged video streaming platform, wherein the sensor data includes an audio stream; The received video stream and sensor data stream are decoded using flv.js, and the decoded audio stream is sent to the WebAudio API interface. The Web Audio API interface analyzes and processes the decoded audio stream to obtain audio sampling information; The pre-created second canvas is used to receive and hide the decoded video stream, and the pre-created third canvas is used to receive the audio sampling information processed by the Web Audio API interface, and to draw and hide the corresponding soundprint. The real-time images of the second and third canvases are copied to the pre-created first canvas frame by frame, pixel by pixel. The first, second, and third canvases are pre-created using the canvas tag in HTML5. The client browser displays real-time information from the first canvas.

2. The method for displaying video by fusing multi-sensor information of a robot according to claim 1, characterized in that: The client browser pre-creates a canvas using the canvas tag in HTML5, which also includes a hidden and displayed fourth canvas. The fourth canvas is used to receive infrared video streams, perform keying on target people or objects in the infrared video streams, and hide and display the keyed images. The real-time image of the fourth canvas is copied onto the first canvas.

3. The method for video display by fusion of multi-sensor information in a robot according to claim 1, characterized in that: The viewing request includes the robot ID, camera ID, and sensor ID.

4. The method for video display by fusion of multi-sensor information in a robot according to claim 1, characterized in that, The converged video streaming platform performs the following steps: Receive the video stream sent by the robot, use FFMPEG software decoding to convert the video stream from RTSP stream to RTMP stream, encapsulate the converted video stream in FLV format and store it in the database; The system receives sensor data sent by the robot, including audio streams. It uses FFMPGE software decoding to convert the audio streams from any format to AAC format. It encapsulates the transcoded audio streams and other sensor data besides the audio streams in Flv format and stores them in the database. The other sensors include gas sensors, temperature sensors, and photoelectric sensors. It receives viewing requests from client browsers, retrieves corresponding video streams and sensor data encapsulated in FLV format from the database based on the requests, and sends them to the client browser, which then creates and displays the merged video stream.

5. A video display device for fusion of multi-sensor information of a robot, characterized in that, Applied to client browsers, including: The customer request sending module is used to create a viewing request and send the viewing request to the integrated video streaming platform mounted on the server. The data receiving module is used to receive video streams and sensor data encapsulated in FLV format transmitted by the converged video streaming platform, wherein the sensor data includes audio streams; The decoding module uses flv.js to decode the video stream and sensor data stream transmitted by the converged video streaming platform, and the decoded audio stream is sent to the Web Audio API interface. The Web Audio API interface module is used to analyze and process the decoded audio stream to obtain audio sampling information; The canvas creation module is used to create canvases using the canvas tag in HTML5. The display module receives and hides the decoded video stream using a pre-created second canvas, receives audio sampling information processed by the Web Audio API using a pre-created third canvas, draws and hides the corresponding soundprint, and copies the real-time images of the second and third canvases to the pre-created first canvas frame by frame, and displays the real-time information on the first canvas.

6. A robot multi-sensor information fusion video display device according to claim 5, characterized in that: The display module also includes a hidden display fourth canvas, which is used to receive infrared video streams, perform keying on target people or objects in the infrared video streams, and display the keyed images in a hidden manner. The real-time image of the fourth canvas is copied onto the first canvas.

7. A robot multi-sensor information fusion video display device according to claim 5, characterized in that, The converged video streaming platform includes: The video stream processing module is used to receive the video stream sent by the robot, and to use FFMPEG software decoding to convert the video stream from RTSP stream to RTMP stream. The transcoded video stream is then encapsulated in FLV format and stored in the database. The audio stream receiving and processing module is used to receive sensor data sent by the robot, including audio streams, and to use FFMPGE software decoding to convert the audio streams from any format to AAC format audio streams. The transcoded audio streams are encapsulated in Flv format, along with other sensor data besides the audio streams, and stored in the database. The other sensors include gas sensors, temperature sensors, and photoelectric sensors. The customer request processing module is used to receive viewing requests initiated by the client browser, and to retrieve the corresponding video stream and sensor data encapsulated in FLV format from the database based on the viewing request, and send them to the client browser, which then creates and displays the fused video stream.

8. A robot multi-sensor information fusion video display system, characterized in that: This includes robots equipped with dynamic cameras and multiple sensors, servers, and one or more client browsers; The sensors are used to detect information about the robot's environment, including one or more of the following: microphone, sound card, pickup, temperature sensor, humidity sensor, positioning device, and harmful gas concentration sensor. The client browser is equipped with a robot multi-sensor information fusion video display device as described in any one of claims 5 to 7.

9. A storage medium, characterized in that, Used to store a computer program; wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 4.

Images