Video data processing method, device, and electronic device

By obtaining and processing image frames from real-time three-dimensional video streams in AR technology and performing three-dimensional registration and rendering, the problem that two-dimensional images cannot reflect the position and posture of objects is solved, and the integration of virtual information and real scenes is improved and the accuracy of recognition is accurate. , realizes efficient real-time display of AR technology.

CN115361576BActive Publication Date: 2024-11-19CHINA TELECOM CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210852428.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-20
Publication Date
2024-11-19
Estimated Expiration
2042-07-20

AI Technical Summary

Technical Problem

In the existing AR technology, two-dimensional images cannot fully reflect the actual position and posture of the object, and cannot judge whether there is mutual occlusion between the objects, resulting in insufficient fusion between virtual information and real scenes and low recognition accuracy.

Method used

By obtaining image frames from the real-time three-dimensional video stream of the target real-life scene, distribute them to different computing nodes for video image processing, obtaining the location and feature descriptors of the target object, performing three-dimensional registration and layer rendering, improving virtual information and real scenes Fusion and identification accuracy.

Benefits of technology

It realizes the efficient integration of virtual information and real scenarios in AR technology application scenarios, improves the accuracy and real-timeness of target object recognition, and ensures the real-time display of virtual information and real information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115361576B_ABST
    Figure CN115361576B_ABST
Patent Text Reader

Abstract

The present application discloses a video data processing method and device, which belongs to the field of AR technology and can improve the integration degree and recognition accuracy of virtual information and real scenes in AR application scenarios. The method includes: obtaining N image frames from the real-time three-dimensional video stream of the target real scene, sending them to different computing nodes for video image processing, and obtaining the target information and position of each target object in each image frame; performing three-dimensional registration of the target information in the corresponding image frame to obtain the corresponding image frame data superimposed with the target information; performing layer rendering according to the image frame data to obtain the rendered image corresponding to the corresponding image frame; and sending the rendered image to the display device for display in the order of the timestamps of the corresponding image frames. This method improves the integration degree of virtual information and real scenes and the recognition accuracy of target objects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of AR technology, and in particular to a video data processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] In the application process of AR (Augmented Reality, AR for short), it is usually necessary to use AR equipment to collect two-dimensional video images of real scenes in real time, identify the target objects in the collected scene images, and then superimpose the identified target information on the real scene images. This AR application can provide individuals and corporate users with fast scanning recognition and real-time AR fusion output capabilities, which helps to achieve the transformation of traditional inspection, supervision and other scenarios to innovative information intuitive display modes. However, two-dimensional images cannot fully reflect the actual position and posture of objects, cannot reflect the depth information in real scenes, and it is difficult to determine whether there is mutual occlusion between objects, resulting in insufficient fusion of virtual information and real scenes and low recognition accuracy in AR technology application scenarios.

[0003] It can be seen that the AR application of video scenes in the existing technology still needs to be improved. Summary of the invention

[0004] The embodiments of the present application provide a video data processing method and device, which can improve the integration of virtual information and real scenes and the accuracy of target recognition in AR technology application scenarios.

[0005] In a first aspect, an embodiment of the present application provides a video data processing method, including:

[0006] Obtain N image frames from a real-time three-dimensional video stream of a target real scene, and send them to different computing nodes for video image processing to obtain target object information and position of each target object in each of the image frames, where N is a natural number greater than 1;

[0007] According to the position, three-dimensionally register the target object information obtained by each computing node in the corresponding image frame to obtain corresponding image frame data superimposed with the target object information;

[0008] Perform layer rendering respectively according to the image frame data to obtain rendered images corresponding to the image frames;

[0009] The rendered image is sent to a designated display device for display in the order of timestamps of corresponding image frames.

[0010] Optionally, the step of acquiring N image frames from a real-time three-dimensional video stream of a target real scene, sending them to different computing nodes for video image processing, and obtaining target object information and a position of each target object in each of the image frames includes:

[0011] Obtain N image frames from a real-time three-dimensional video stream of a target real scene;

[0012] Sending each of the image frames to different computing nodes for video image processing to obtain target feature descriptors and positions of each target object in the corresponding image frame;

[0013] For each of the image frames, the target feature descriptors of each target in the image frame are matched and identified based on a preset target feature descriptor database to obtain target information of each target in the image frame.

[0014] Optionally, acquiring N image frames from a real-time three-dimensional video stream of a target real scene includes:

[0015] Through distributed data processing technology, the video image processing task of the real-time three-dimensional video stream of the target real scene is decomposed to obtain N image frames.

[0016] Optionally, by using distributed data processing technology, the video image processing task of the real-time three-dimensional video stream of the target real scene is decomposed to obtain N image frames, including:

[0017] Collect video data of N image frames from a real-time three-dimensional video stream of a target real scene;

[0018] Independently encapsulate the video data of each image frame into a binary video stream, and send it to a distributed message queue;

[0019] Each of the binary video streams is obtained from the distributed message queue, and a corresponding image frame is generated according to the binary video stream.

[0020] Optionally, acquiring each binary video stream from the distributed message queue and generating a corresponding image frame according to the binary video stream includes:

[0021] Using the data execution engine Spark to obtain the binary video stream from the distributed message queue;

[0022] The binary video stream is decoded to generate corresponding image frames.

[0023] Optionally, the preset target object feature descriptor database is pre-established offline, and the preset target object feature descriptor database includes: an association relationship between a target feature descriptor of a target object and target object information.

[0024] In a second aspect, an embodiment of the present application provides a video data processing device, including:

[0025] A target object information and position acquisition module, used to acquire N image frames from a real-time three-dimensional video stream of a target real scene, and send them to different computing nodes for video image processing to obtain target object information and position of each target object in each of the image frames, wherein N is a natural number greater than 1;

[0026] A target object information superposition module, used to perform three-dimensional registration on the target object information obtained by each computing node in the corresponding image frame according to the position, to obtain corresponding image frame data superimposed with the target object information;

[0027] A rendering module, used to perform layer rendering according to the image frame data to obtain a rendered image corresponding to the image frame;

[0028] The display module is used to send the rendered image to a designated display device for display according to the time stamp sequence of the corresponding image frames.

[0029] Optionally, the target object information and position acquisition module is further used to:

[0030] Obtain N image frames from a real-time three-dimensional video stream of a target real scene;

[0031] Sending each of the image frames to different computing nodes for video image processing to obtain target feature descriptors and positions of each target object in the corresponding image frame;

[0032] For each of the image frames, the target feature descriptors of each target in the image frame are matched and identified based on a preset target feature descriptor database to obtain target information of each target in the image frame.

[0033] Optionally, acquiring N image frames from a real-time three-dimensional video stream of a target real scene includes:

[0034] Through distributed data processing technology, the video image processing task of the real-time 3D video stream of the target real scene is decomposed to obtain N image frames

[0035] Optionally, by using distributed data processing technology, the video image processing task of the real-time three-dimensional video stream of the target real scene is decomposed to obtain N image frames, including:

[0036] Collect video data of N image frames from a real-time three-dimensional video stream of a target real scene;

[0037] Independently encapsulate the video data of each image frame into a binary video stream, and send it to a distributed message queue;

[0038] Each of the binary video streams is obtained from the distributed message queue, and a corresponding image frame is generated according to the binary video stream.

[0039] In some embodiments of the present application, obtaining each binary video stream from the distributed message queue and generating a corresponding image frame according to the binary video stream includes:

[0040] Using the data execution engine Spark to obtain the binary video stream from the distributed message queue;

[0041] The binary video stream is decoded to generate corresponding image frames.

[0042] In some embodiments of the present application, the preset target object feature descriptor database is pre-established offline, and the preset target object feature descriptor database includes: an association relationship between a target feature descriptor of a target object and target object information.

[0043] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video data processing method described in the embodiment of the present application when executing the computer program.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the video data processing method disclosed in the embodiment of the present application are performed.

[0045] The video data processing method disclosed in the embodiment of the present application obtains N image frames from a real-time three-dimensional video stream of a target real scene, and sends them to different computing nodes for video image processing to obtain target information and positions of each target object in each image frame, wherein N is a natural number greater than 1; according to the positions, the target information obtained by each computing node is three-dimensionally registered in the corresponding image frame to obtain corresponding image frame data superimposed with the target information; layer rendering is performed according to the image frame data to obtain rendered images corresponding to the corresponding image frames; and the rendered images are sent to a designated display device for display in the order of timestamps of the corresponding image frames. This can improve the degree of integration of virtual information and real scenes and the accuracy of target recognition in the application scenarios of AR technology, and can ensure the real-time integration and display of real scene information and virtual information.

[0046] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] Figure 1 It is one of the flow charts of the video data processing method in the embodiment of the present application;

[0049] Figure 2 is a schematic diagram of a system architecture for implementing a video data processing method in an embodiment of the present application;

[0050] Figure 3 is a schematic diagram of an application scenario of the video data processing method in an embodiment of the present application;

[0051] Figure 4 is a schematic diagram of the structure of a video data processing device in an embodiment of the present application;

[0052] Figure 5 A block diagram schematically shows an electronic device for executing the method according to the present application; and

[0053] Figure 6 A storage unit for holding or carrying a program code for implementing the method according to the present application is schematically shown. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] Embodiment 1

[0056] like Figure 1 As shown, a video data processing method disclosed in an embodiment of the present application includes: steps 100 to 140.

[0057] Step 110, obtain N image frames from the real-time three-dimensional video stream of the target real scene, and send them to different computing nodes for video image processing to obtain the target object information and position of each target object in each of the image frames.

[0058] Wherein, N is a natural number greater than 1.

[0059] The target reality scene described in the embodiments of the present application is a scene for scene display based on the collected video images, including but not limited to: interactive display and training, smart supervision, digital inspection and other scenes.

[0060] As three-dimensional data acquisition equipment becomes more and more common, applying the results of identifying objects in three-dimensional images to AR technology can improve the accuracy of obtaining target information, as well as help the scalability of applications. In an embodiment of the present application, video data processing is performed based on a real-time three-dimensional video stream of a target reality scene to obtain richer information from the original video image, such as extracting information such as the geometry, structure, color, and coordinates of the target object to improve the accuracy of identifying the target object in the video image, and to expand richer AR applications. Therefore, in an embodiment of the present application, video image frames are sampled from a real-time three-dimensional video stream of a target reality scene for subsequent display of virtual-real fusion scene information.

[0061] The real-time 3D video stream described in the embodiment of the present application can be a video stream collected by a 3D video image acquisition device deployed in the target real scene, or a 2D video stream collected by multiple 2D images deployed in the target real scene, and then the 2D video stream is converted into a point cloud to obtain a 3D video stream. In the embodiment of the present application, the method for obtaining the real-time 3D video stream is not limited.

[0062] The N image frames may be image frames sampled from a real-time 3D video stream of a target real scene at a preset frequency. For example, if a real-time 3D video stream of a computer room for digital inspection is a video stream that collects 30 frames of video images per second, 3 image frames may be collected from the 30 frames of video images at a frequency of 3 frames per second.

[0063] Afterwards, the collected N frames of images are sent to different computing nodes for video image processing to obtain the target information and position of each target in each image frame. The video image processing includes but is not limited to: identifying each target included in each image frame, obtaining the position coordinates of each target in the corresponding image, and the target information matched by each target.

[0064] In the embodiment of the present application, the target object information is determined according to the specific application, and the target object information categories of different targets can be the same or different. For example, for a computer room scene, the target object can be a server, and correspondingly, the target object information can be information such as the model, performance, and current usage status of the server.

[0065] Three-dimensional object recognition is usually based on three-dimensional point cloud data. Since three-dimensional point cloud data includes the geometry, structure, color, coordinates and other combined information of the object, there is a huge amount of data. In the prior art, AR applications based on three-dimensional images have the defects of slow calculation speed, low recognition efficiency and poor real-time performance. In the embodiment of the present application, in order to improve the real-time performance of AR applications, that is, to improve the real-time performance of superimposing and displaying target information in the real-time video stream of the real scene, N image frames are sent to N computing nodes for video image processing, and each computing node performs video image processing on one image frame to obtain the target information and position of each target object in the image frame, thereby improving the processing speed of N frames of video images.

[0066] Step 120 , according to the position, three-dimensionally register the target object information obtained by each computing node in the corresponding image frame to obtain corresponding image frame data superimposed with the target object information.

[0067] Three-dimensional registration refers to obtaining the accurate coordinates of specific objects in three-dimensional space through the analysis process of computer graphics, and then splicing the computer-generated virtual objects into the real space based on the obtained coordinates, so that the real environment and virtual objects can be accurately and seamlessly integrated.

[0068] In an embodiment of the present application, after obtaining the target information and position of each target object in the image frame by performing video image processing on the corresponding image frame by each computing node, for each image frame, the target information of each target object in the image frame is further superimposed near the position of the target object in the image to obtain the corresponding image frame data superimposed with the target information, so as to achieve a display effect of displaying the target information of the target object near the corresponding target object when the image is displayed based on the corresponding image frame data superimposed with the target information. The target information can move with the target object, realizing seamless fusion of the real environment and the target information (i.e., virtual information).

[0069] Step 130 , performing layer rendering according to the image frame data to obtain rendered images corresponding to the image frames.

[0070] Next, layer rendering is performed according to the corresponding image frame data superimposed with the target object information to obtain a rendered image corresponding to the corresponding image frame. For example, the bottom layer image rendering is performed according to the original image data in the image frame data, and the top layer image rendering is performed according to the target object information in the image frame data, so as to obtain a rendered image corresponding to the corresponding image frame.

[0071] In some embodiments of the present application, the image frame data may also include data of multiple layers, and the data of different layers correspond to the image data of objects at different depths in the three-dimensional image. Further, image rendering may be performed in sequence according to the data of different layers, thereby obtaining a rendered image corresponding to the image frame.

[0072] According to this method, after N image frames are fused with target object data respectively, N frames of image frame data are obtained. After layered rendering is performed on each frame of image frame data, N rendered images can be obtained, wherein each rendered image corresponds to an image frame.

[0073] Step 140: Send the rendered image to a designated display device for display in the order of the timestamps of the corresponding image frames.

[0074] After the above steps, a rendered image corresponding to each of the N image frames can be obtained. Next, the rendered images corresponding to each of the N image frames are sent to the designated display device for display in sequence according to the order of the timestamps of the N image frames. For example, the rendered images corresponding to each of the image frames are sent to the AR device for display in sequence.

[0075] In order to facilitate readers to understand the video data processing method in the AR application disclosed in the embodiments of the present application, the specific implementation method of obtaining the target object information and position from the image frame is further explained below.

[0076] In some embodiments of the present application, N image frames are obtained from a real-time three-dimensional video stream of a target real scene, and are sent to different computing nodes for video image processing to obtain target information and positions of each target object in each image frame, including: obtaining N image frames from a real-time three-dimensional video stream of a target real scene; sending each of the image frames to different computing nodes for video image processing to obtain target feature descriptors and positions of each target object in the corresponding image frames; for each of the image frames, based on a preset target feature descriptor database, matching and identifying the target feature descriptors of each target object in the image frame to obtain target information of each target object in the image frame.

[0077] The specific implementation method of obtaining N image frames from the real-time three-dimensional video stream of the target real scene is as described above and will not be repeated here.

[0078] After each of the image frames is sent to different computing nodes for video image processing, each of the computing nodes performs video image processing on a frame of the image frame to obtain the target feature descriptor and position of each target object in the corresponding image frame. Wherein, the target feature descriptor is a vector representation that describes the target without features. In some embodiments of the present application, after the computing node identifies the target object in the image frame, it first determines the position coordinates of the target object in the image frame, and then the normal vector angle difference and distance difference of every two points in the neighborhood of the key point of the target object can be statistically calculated using a histogram, and the feature vector calculated is used as the target feature descriptor of the target object. In this way, for each image frame, the computing node can obtain the target feature descriptor of each target object therein.

[0079] In other embodiments of the present application, other methods may be used to calculate the target feature descriptor. In the embodiments of the present application, the specific method for identifying the target in the video image frame is not limited, and the specific implementation method for obtaining the target feature descriptor of the target object is not limited.

[0080] In some embodiments of the present application, before obtaining N image frames from a real-time three-dimensional video stream of a target real scene and sending them to different computing nodes for video image processing to obtain the target information and position of each target in each of the image frames, it also includes: establishing a target feature descriptor database offline. The target feature descriptor database includes: the association relationship between the target feature descriptor and the target information of the target.

[0081] For example, by pre-collecting images of targets that may appear in video images, and using the aforementioned method to obtain the target feature descriptors of the targets in each image, the target feature descriptors of each target are then associated with the target information of the corresponding target and stored in a database as offline data for subsequent matching.

[0082] After the target information of each target object in each of the image frames is matched, for each of the image frames, the target feature descriptor of each target object included therein (for example, denoted as C_cur) can be used as the target feature descriptor to be identified, and matched with the target feature descriptor in the preset target feature descriptor database (for example, denoted as Ci), and the target information associated with the successfully matched target feature descriptor in the preset target feature descriptor database (such as the aforementioned Ci) is used as the target information of the target object corresponding to the target feature descriptor to be identified (such as the aforementioned C_cur).

[0083] In some embodiments of the present application, in order to ensure the real-time performance of AR applications, the mainstream J2EE development standards, microservice architecture, and containerized hosting are adopted to support cross-platform and cross-database applications. By calling the function interface in the Open CV library (an open source interface library) and the point cloud PCL library, the services, plug-ins, and middleware (such as Tomcat, Nginx, Mycat, Mysql, and ElasticSearch) are containerized and deployed in combination with docker. The overall algorithm technology route is advanced, feasible, reasonable, and mature.

[0084] In some embodiments of the present application, the step of obtaining N image frames from a real-time three-dimensional video stream of a target real scene includes: decomposing a video image processing task of the real-time three-dimensional video stream of the target real scene by distributed data processing technology to obtain N image frames. The distributed data processing technology includes: real-time data stream processing technology and data transmission technology.

[0085] In some embodiments of the present application, the video image processing task of the real-time three-dimensional video stream of the target real scene is decomposed by distributed data processing technology to obtain N image frames, including: collecting video data of N image frames from the real-time three-dimensional video stream of the target real scene; independently encapsulating the video data of each image frame into a binary video stream and sending it to a distributed message queue; obtaining each binary video stream from the distributed message queue, and generating corresponding image frames according to the binary video stream. For example, each image frame can be used as a basic unit, and the image data of each image frame can be encapsulated into a binary video stream and sent to a kafka (a distributed message queue) distributed message queue. By encapsulating each image frame into a binary video stream and distributing it independently, it is convenient for subsequent independent reading, thereby achieving distributed processing.

[0086] In some embodiments of the present application, the step of obtaining each binary video stream from the distributed message queue and generating corresponding image frames according to the binary video stream includes: obtaining the binary video stream from the distributed message queue using the data execution engine Spark; and generating corresponding image frames according to the decoding of the binary video stream. For example, using the Spark Streaming framework (a real-time video streaming technology), the data execution engine Spark obtains binary video stream data from the Kafka end, and then decodes the obtained binary video stream data into a picture, i.e., a video image. By accelerating the distribution and calculation of video streams using Spark technology, it is helpful to improve the speed of target recognition, thereby improving the real-time performance of AR applications.

[0087] Combine the following Figure 2The video data processing method shown is a schematic diagram of a system architecture for implementing the method, which further illustrates the implementation method of the method.

[0088] like Figure 2 As shown, the implementation system architecture includes five modules: data acquisition, data storage, data processing, data service, and business application. The following examples illustrate the role of each module in the video data processing method.

[0089] The data acquisition module uses Flume to collect, aggregate, transmit and process massive distributed data. For example, when establishing a preset target feature descriptor database offline, it collects, aggregates and transmits target images through AR devices deployed in various scenes, and in the specific application process, it collects, aggregates and transmits real-time 3D video streams of various real scenes.

[0090] The data storage module provides storage services for distributed columnar data through data storage technologies such as Mysql and HDFS. The Sqoop tool transfers data between Hive and Mysql. The Mq (message queue) message technology is used to solve problems such as traffic peak shaving, and achieve high performance, high availability, scalability and eventual consistency architecture.

[0091] The data processing module is used for offline data processing and real-time data processing. For example, the target feature descriptor is extracted from the image of the target object collected offline, and for another example, the target object is identified and the target object information is obtained from the image frame of the target real scene collected. The data processing module uses the MapReduce computing model to decompose large data processing tasks, quickly and parallelly process a large amount of data scanned by the front end, and uses the Spark data analysis engine to efficiently process data streams based on memory, so as to achieve the scalability and high throughput of real-time data and ensure the real-time display. At the same time, the data processing module uses Spark Streaming to match the real-time data (temporarily stored in Hive) with the offline data (data stored in mysql, hive and mysql) to complete the target object identification (determine the location and target object information of the target object in the image frame).

[0092] The data service module is used to receive the results obtained after data processing by the data processing module and provide a service interface for the business application module.

[0093] The business application module is used to implement AR applications, such as layer rendering, display, and AR interaction.

[0094] In some embodiments of the present application, the data acquisition module, data storage module, data processing module, and data service module are usually deployed on the server side, and the business application module is usually deployed on the terminal side, such as the display device side. Figure 3 Take the AR application in the interactive display and training scene shown as an example, wherein the AR glasses 310 are used to collect the real-time three-dimensional video stream of the target real scene, and the AR glasses 310 and the smart phone 330 are used as display devices to display the video stream integrated with the target object information in the target real scene in real time. During the display or training, the introducer wears the AR glasses 310, collects the real-time three-dimensional video stream of the target real scene through the AR glasses 310, and sends the collected real-time three-dimensional video stream to the server 320; then, the server 320 obtains N image frames from the real-time three-dimensional video stream of the target real scene, and sends them to different computing nodes for video image processing, and obtains the target object information and position of each target object in each of the image frames, wherein N is a natural number greater than 1, and according to the position, the target object information obtained by each computing node is respectively registered in the corresponding image frame in three dimensions, and the corresponding image frame data superimposed with the target object information is obtained, and then, according to the image frame data, the layer rendering is performed respectively to obtain the rendered image corresponding to the corresponding image frame, and the rendered image is sent to the smart phone 330 for display according to the time stamp sequence of the corresponding image frame. At the same time, the server 320 can also send the rendered image to the AR glasses for display.

[0095] By adopting the video data processing method disclosed in the embodiments of the present application, it is possible to achieve a demonstration or training in which the introducer wears AR glasses to scan and identify objects in the real scene in front of him, and the viewer's terminal displays the first-person perspective image superimposed with virtual video, graphic introduction and other information in real time. Compared with simple video images or file exchanges, it is more propaganda-appealing and easier for viewers to grasp the information about the introduced objects.

[0096] The video data processing method disclosed in the embodiment of the present application can also be applied to scenarios such as smart bank supervision. For example, bank staff can wear AR glasses in the computer room and scan cabinet equipment to read the corresponding real-time IoT data, and display it in the real-time picture by integrating the real scene at the viewing end, thus realizing the transformation from traditional supervision to intelligent supervision mode with intuitive information display, reducing security risks, reducing labor costs, and improving enterprise decision-making efficiency.

[0097] For another example, when the video data processing method disclosed in the embodiment of the present application is used by employees in a digital inspection scene, the inspectors wear AR glasses and scan the on-site images to obtain relevant information about objects on the scene. The inspection and monitoring screen and AR information can be displayed in a first-person perspective live video recording, forming a closed loop of the entire process of on-site inspection and review. Intelligent AR recognition and inspection can break through the limitations of space and time, free your hands, and ensure the safety of the work site. Complete detailed inspections of items such as inspection personnel on duty, production operations, and material support in a short time.

[0098] The above AR application scenarios are only some of the application scenarios listed for the convenience of readers to understand, and are not all application scenarios of the video data processing method disclosed in the embodiments of this application. Those skilled in the art should understand that the video data processing method disclosed in the embodiments of this application can also be applied in other scenarios.

[0099] The video data processing method disclosed in the embodiment of the present application obtains N image frames from a real-time three-dimensional video stream of a target real scene, and sends them to different computing nodes for video image processing to obtain target information and positions of each target object in each image frame, wherein N is a natural number greater than 1; according to the positions, the target information obtained by each computing node is three-dimensionally registered in the corresponding image frame to obtain corresponding image frame data superimposed with the target information; layer rendering is performed according to the image frame data to obtain rendered images corresponding to the corresponding image frames; and the rendered images are sent to a designated display device for display in the order of timestamps of the corresponding image frames. This can improve the degree of integration of virtual information and real scenes and the accuracy of target recognition in the application scenarios of AR technology, and can ensure the real-time integration and display of real scene information and virtual information.

[0100] The video data processing method disclosed in the embodiment of the present application can obtain richer information about the target in the real scene by collecting the actual three-dimensional video of the real scene for use in AR applications, thereby improving the integration of virtual information and real scenes and the accuracy of target recognition. Furthermore, by sending the three-dimensional video image frames to different computing nodes for parallel distributed video image processing, it can effectively solve the problem that due to the large amount of three-dimensional video image data and the slow processing speed of traditional data processing methods, the real-time fusion display of virtual information and real scenes cannot be met, and the real-time fusion display of real scene information and virtual information can be effectively improved. The video data processing method disclosed in the embodiment of the present application provides technical support for improving the interactivity of AR technology and satisfying a higher user experience.

[0101] Furthermore, in an embodiment of the present application, by encapsulating real-time video into a binary data stream that can be processed in parallel and distributed for distribution and transmission, through a distributed processing algorithm and using Spark data analysis engine technology, the data stream computing tasks are decomposed and distributed to other nodes for processing, and the data is imported into memory for calculation, it is possible to ensure real-time performance in high-computation scenarios such as AR real-time recognition of three-dimensional objects and display of layer information.

[0102] Embodiment 2

[0103] The embodiment of the present application discloses a video data processing device, such as Figure 4 As shown, the device comprises:

[0104] The target object information and position acquisition module 410 is used to acquire N image frames from the real-time three-dimensional video stream of the target real scene, and send them to different computing nodes for video image processing to obtain the target object information and position of each target object in each of the image frames, wherein N is a natural number greater than 1;

[0105] The target object information superposition module 420 is used to perform three-dimensional registration on the target object information obtained by each computing node in the corresponding image frame according to the position, so as to obtain corresponding image frame data superimposed with the target object information;

[0106] A rendering module 430 is used to perform layer rendering according to the image frame data to obtain a rendered image corresponding to the image frame;

[0107] The display module 440 is used to send the rendered image to a designated display device for display according to the time stamp sequence of the corresponding image frames.

[0108] In some embodiments of the present application, the target object information and position acquisition module 410 is further used to:

[0109] Obtain N image frames from a real-time three-dimensional video stream of a target real scene;

[0110] Sending each of the image frames to different computing nodes for video image processing to obtain target feature descriptors and positions of each target object in the corresponding image frame;

[0111] For each of the image frames, the target feature descriptors of each target in the image frame are matched and identified based on a preset target feature descriptor database to obtain target information of each target in the image frame.

[0112] In some embodiments of the present application, acquiring N image frames from a real-time three-dimensional video stream of a target real scene includes:

[0113] Through distributed data processing technology, the video image processing task of the real-time three-dimensional video stream of the target real scene is decomposed to obtain N image frames.

[0114] In some embodiments of the present application, the video image processing task of the real-time three-dimensional video stream of the target real scene is decomposed by distributed data processing technology to obtain N image frames, including:

[0115] Collect video data of N image frames from a real-time three-dimensional video stream of a target real scene;

[0116] Independently encapsulate the video data of each image frame into a binary video stream, and send it to a distributed message queue;

[0117] Each of the binary video streams is obtained from the distributed message queue, and a corresponding image frame is generated according to the binary video stream.

[0118] In some embodiments of the present application, obtaining each binary video stream from the distributed message queue and generating a corresponding image frame according to the binary video stream includes:

[0119] Using the data execution engine Spark to obtain the binary video stream from the distributed message queue;

[0120] The binary video stream is decoded to generate corresponding image frames.

[0121] In some embodiments of the present application, the preset target object feature descriptor database is pre-established offline, and the preset target object feature descriptor database includes: an association relationship between a target feature descriptor of a target object and target object information.

[0122] The video data processing device disclosed in the embodiment of the present application is used to implement the video data processing method described in the first embodiment of the present application. The specific implementation methods of each module of the device will not be repeated here. Please refer to the specific implementation methods of the corresponding steps in the method embodiment.

[0123] The video data processing device disclosed in the embodiment of the present application obtains N image frames from a real-time three-dimensional video stream of a target real scene, and sends them to different computing nodes for video image processing to obtain target information and positions of each target object in each image frame, wherein N is a natural number greater than 1; according to the positions, the target information obtained by each computing node is three-dimensionally registered in the corresponding image frame to obtain corresponding image frame data superimposed with the target information; layer rendering is performed according to the image frame data to obtain rendered images corresponding to the corresponding image frames; and the rendered images are sent to a designated display device for display in the order of timestamps of the corresponding image frames. This can improve the degree of integration of virtual information and real scenes and the accuracy of target recognition in the application scenario of AR technology, and can ensure the real-time integration and display of real scene information and virtual information.

[0124] The video data processing device disclosed in the embodiment of the present application can obtain richer information about the target object in the real scene by collecting the actual three-dimensional video of the real scene for use in AR applications, thereby improving the integration degree and recognition accuracy of virtual information and real scenes. Furthermore, by sending the three-dimensional video image frames to different computing nodes for parallel distributed video image processing, it can effectively solve the problem that due to the large amount of three-dimensional video image data and the slow processing speed of the traditional data processing method, the real-time fusion display of virtual information and real scenes cannot be met, and the real-time fusion display of real scene information and virtual information can be effectively improved. The video data processing method disclosed in the embodiment of the present application provides technical support for improving the interactivity of AR technology and satisfying a higher user experience.

[0125] Furthermore, in an embodiment of the present application, by encapsulating real-time video into a binary data stream that can be processed in parallel and distributed for distribution and transmission, through a distributed processing algorithm and using Spark data analysis engine technology, the data stream computing tasks are decomposed and distributed to other nodes for processing, and the data is imported into memory for calculation, it is possible to ensure real-time performance in high-computation scenarios such as AR real-time recognition of three-dimensional objects and display of layer information.

[0126] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0127] The above is a detailed introduction to a video data processing method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and a core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0129] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the electronic device according to the embodiment of the present application. The present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0130] For example, Figure 5 An electronic device that can implement the method according to the present application is shown. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. The electronic device traditionally includes a processor 510 and a memory 520 and a program code 530 stored on the memory 520 and can be run on the processor 510, and the processor 510 implements the method described in the above embodiment when executing the program code 530. The memory 520 can be a computer program product or a computer-readable medium. The memory 520 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. The memory 520 has a storage space 5201 for the program code 530 of the computer program for executing any method step in the above method. For example, the storage space 5201 for the program code 530 can include various computer programs for implementing the various steps in the above method respectively. The program code 530 is a computer-readable code. These computer programs can be read from one or more computer program products or written into the one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. The computer program includes a computer readable code, and when the computer readable code is executed on an electronic device, the electronic device is caused to execute the method according to the above embodiment.

[0131] The embodiment of the present application further discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the video data processing method as described in the first embodiment of the present application are implemented.

[0132] Such a computer program product may be a computer-readable storage medium, which may have a computer-readable storage medium having a Figure 5The memory 520 in the electronic device shown in the figure is similar to the storage segment, storage space, etc. The program code can be compressed and stored in the computer-readable storage medium in an appropriate form. The computer-readable storage medium is generally as shown in the reference Figure 6 The portable or fixed storage unit. Generally, the storage unit includes computer readable code 530', which is a code read by a processor. When the code is executed by the processor, each step in the method described above is implemented.

[0133] The term "one embodiment", "embodiment" or "one or more embodiments" herein means that a particular feature, structure or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present application. In addition, please note that the examples of the term "in one embodiment" here do not necessarily all refer to the same embodiment.

[0134] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0135] In the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video data processing method, It is characterized in that include: Obtain N image frames from a real-time three-dimensional video stream of a target real scene, and send them to different computing nodes for video image processing to obtain target object information and position of each target object in each of the image frames, where N is a natural number greater than 1; According to the position, three-dimensionally register the target object information obtained by each computing node in the corresponding image frame to obtain corresponding image frame data superimposed with the target object information; Perform layer rendering respectively according to the image frame data to obtain rendered images corresponding to the image frames; Sending the rendered image to a designated display device for display in a chronological order of timestamps of corresponding image frames; The step of acquiring N image frames from a real-time three-dimensional video stream of a target real scene includes: By using distributed data processing technology, video data of N image frames are collected from the real-time three-dimensional video stream of the target real scene; Independently encapsulate the video data of each image frame into a binary video stream, and send it to a distributed message queue; Each of the binary video streams is obtained from the distributed message queue, and a corresponding image frame is generated according to the binary video stream to obtain N image frames.

2. The method according to claim 1, It is characterized in that The step of acquiring N image frames from the real-time three-dimensional video stream of the target real scene, sending them to different computing nodes for video image processing, and obtaining the target object information and position of each target object in each of the image frames includes: Obtain N image frames from a real-time three-dimensional video stream of a target real scene; Sending each of the image frames to different computing nodes for video image processing to obtain target feature descriptors and positions of each target object in the corresponding image frame; For each of the image frames, the target feature descriptors of each target in the image frame are matched and identified based on a preset target feature descriptor database to obtain target information of each target in the image frame.

3. The method according to claim 1, It is characterized in that The acquiring each binary video stream from the distributed message queue and generating a corresponding image frame according to the binary video stream includes: Using the data execution engine Spark to obtain the binary video stream from the distributed message queue; The binary video stream is decoded to generate corresponding image frames.

4. The method according to claim 2, It is characterized in that The preset target object feature descriptor database is pre-established offline, and the preset target object feature descriptor database includes: the association relationship between the target feature descriptor of the target object and the target object information.

5. A video data processing device, It is characterized in that include: A target object information and position acquisition module, used to acquire N image frames from a real-time three-dimensional video stream of a target real scene, and send them to different computing nodes for video image processing to obtain target object information and position of each target object in each of the image frames, wherein N is a natural number greater than 1; A target object information superposition module, used to perform three-dimensional registration on the target object information obtained by each computing node in the corresponding image frame according to the position, to obtain corresponding image frame data superimposed with the target object information; A rendering module, used to perform layer rendering according to the image frame data to obtain a rendered image corresponding to the image frame; A display module, used for sending the rendered image to a designated display device for display in the order of timestamps of corresponding image frames; The step of acquiring N image frames from a real-time three-dimensional video stream of a target real scene includes: By using distributed data processing technology, video data of N image frames are collected from the real-time three-dimensional video stream of the target real scene; Independently encapsulate the video data of each image frame into a binary video stream, and send it to a distributed message queue; Each of the binary video streams is obtained from the distributed message queue, and a corresponding image frame is generated according to the binary video stream to obtain N image frames.

6. The device according to claim 5, It is characterized in that The target object information and position acquisition module is further used to: Obtain N image frames from a real-time three-dimensional video stream of a target real scene; Sending each of the image frames to different computing nodes for video image processing to obtain target feature descriptors and positions of each target object in the corresponding image frame; For each of the image frames, the target feature descriptors of each target in the image frame are matched and identified based on a preset target feature descriptor database to obtain target information of each target in the image frame.

7. An electronic device comprising a memory, a processor, and a program code stored in the memory and executable on the processor, It is characterized in that When the processor executes the program code, the video data processing method according to any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium having program code stored thereon, It is characterized in that When the program code is executed by a processor, the steps of the video data processing method described in any one of claims 1 to 4 are implemented.