Video-based virtual-real scene mapping method and system, and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHENGDU UNIV OF INFORMATION TECH
- Filing Date
- 2023-11-06
- Publication Date
- 2026-06-02
Smart Images
Figure CN117640884B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of digital twin technology and computer vision technology, and in particular to a video-based method, system, and storage medium for mapping virtual and real scenes. Background Technology
[0002] Currently, video surveillance can only identify target locations by presenting them at fixed angles on the terminal. Therefore, it lacks an intuitive 3D scene perception and interactive control interface for the monitored scene. As the application of video surveillance becomes increasingly widespread across various scenarios, the demand from users for the accuracy and speed of information acquired through monitoring is constantly increasing. Based on the current development of digital twins in China, a key aspect is virtual-real mapping, a prerequisite for achieving physical-to-virtual simulation of the physical world. Simultaneously, different industries are closely following policy and carrying out data mapping work related to digital twins. In the industrial production sector, a large number of sensors and personnel are used to collect and process various environmental indicators and personnel information within factories and construction sites to improve the production efficiency of industrial parks and factories.
[0003] Currently, known classroom monitoring systems use YOLOv5-based detection systems to identify abnormal student behavior, but have not yet implemented video-to-real-video mapping or built twin models. Summary of the Invention
[0004] This invention provides a video-based virtual-real scene mapping method, system, and storage medium, which can identify people and objects in a physical scene in real time (near real time) and map them to their corresponding locations in a virtual twin scene.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] Video-based methods for mapping virtual and real scenes include:
[0007] S1. Obtain surveillance video;
[0008] S2. The surveillance video is processed using the YOLOv5 target recognition and detection algorithm to obtain the location and category information of people and objects in the physical scene, and then transmitted to the MySQL database; the MySQL database is connected through Unity3d, and the TCP network transmission protocol is used to realize the transmission of data across Python and Unity;
[0009] S3. Establish a one-to-one correspondence function relationship between the real scene coordinates and the twin scene coordinates to map people and objects into the corresponding twin scene.
[0010] In some embodiments, in S2, the surveillance video is first uploaded to a streaming media server. YOLOv5 is used to process the real-time uploaded surveillance video for recognition. Based on the previously trained model and the set confidence threshold, objects in the image whose detection boxes reach the threshold are further recognized. The recognized data is then loaded into a MySQL database. Before setting the confidence parameter, classroom images are tested. The recall rate, accuracy, false positive rate, and average false positive rate are considered to set the optimal confidence value according to different classroom environments, achieving the best recognition effect. Then, the mapping fit value is combined with the adjustment to achieve the best mapping effect.
[0011] In some embodiments, during S2, when transmitting the location and category information of people and objects in the physical scene to the MySQL database, the distance from the detected target to the camera is first obtained. In YOLOv5, distance measurement requires first obtaining the pixel coordinates of the target object and its actual size information in the real world. The method for obtaining this information is as follows:
[0012] Obtain the pixel coordinates of the target in the image;
[0013] Obtain the size of the target in the real world: Obtain the size of the target in the real world manually or with other general measuring tools and equipment, using the length of the object when it is at rest as the target size;
[0014] Obtain camera parameters: including the camera's focal length, optical center, and horizontal field of view.
[0015] Distance calculation using the principle of triangle similarity: Assuming the actual size of the target is known, a triangle can be constructed using the target's pixel coordinates and camera parameters to calculate the distance from the target to the camera. The specific formula is as follows:
[0016]
[0017] Where D is the detection distance, F represents the size of the target being detected, W represents the camera focal length, and h is the height of the detection box.
[0018] In some embodiments, in S3, by establishing a one-to-one correspondence between the twin system coordinate system and the physical world coordinate system, the detection distance is mapped to the front-back distance in the twin system, and the mapping formula is:
[0019] y = kx + b;
[0020] Where y is the vertical distance in the twin classroom, x is the detection distance, k is the slope, and b is a constant; the horizontal mapping is a linear mapping in one variable through the x-coordinate of the center coordinate and the horizontal distance of the twin classroom. Due to the perspective effect, the horizontal mapping, compared with the vertical mapping, needs to determine the slope k and constant b through the y-coordinate of the center coordinate, which is also a linear mapping in one variable.
[0021] A video-based virtual-real scene mapping system is used to implement the video-based virtual-real scene mapping method described above.
[0022] The video-based virtual-real scene mapping system includes:
[0023] The video monitoring module is used to acquire real-time video information of the physics classroom through camera sensors;
[0024] The data transmission module is used to transmit video data from video surveillance to the streaming media server via the UDP network transmission protocol, and to realize bidirectional data transmission between the client and the server via Socket communication.
[0025] The data processing and storage module is used to process the video data from the streaming media server using the YOLOv5 object detection algorithm, obtain the category information and location data of people and objects in the classroom, and store the data in the MySQL database.
[0026] The data mapping module is used to map the location data of objects in the MySQL database to the corresponding twin scene.
[0027] A computer-readable storage medium stores computer instructions, which, when read by a computer, execute the aforementioned video-based virtual-real scene mapping method.
[0028] In summary, the present invention has at least the following beneficial effects:
[0029] This invention enables real-time (near real-time) mapping of information about people or objects in a physical scene to a corresponding twin scene, thereby promoting the realization of scene digitization. This mapping method is the foundation for the study of twin scenes and lays the groundwork for subsequent research such as the analysis and operation of twin scenes. Attached Figure Description
[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1 This is a schematic diagram illustrating the principle of the video-based virtual-real scene mapping method involved in this invention.
[0032] Figure 2 This is a flowchart illustrating the video-based virtual-real scene mapping method involved in this invention. Detailed Implementation
[0033] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.
[0034] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.
[0035] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0036] like Figure 1 and Figure 2 As shown, this embodiment provides a video-based virtual-real scene mapping method, including:
[0037] S1. Obtain surveillance video;
[0038] S2. The surveillance video is processed using the YOLOv5 target recognition and detection algorithm to obtain the location and category information of people and objects in the physical scene, and then transmitted to the MySQL database; the MySQL database is connected through Unity3d, and the TCP network transmission protocol is used to realize the transmission of data across Python and Unity;
[0039] S3. Establish a one-to-one correspondence function relationship between the real scene coordinates and the twin scene coordinates to map people and objects into the corresponding twin scene.
[0040] In some embodiments, in S2, the surveillance video is first uploaded to a streaming media server. YOLOv5 is used to process the real-time uploaded surveillance video for recognition. Based on the previously trained model and the set confidence threshold, objects in the image whose detection boxes reach the threshold are further recognized. The recognized data is then loaded into a MySQL database. Before setting the confidence parameter, classroom images are tested. The recall rate, accuracy, false positive rate, and average false positive rate are considered to set the optimal confidence value according to different classroom environments, achieving the best recognition effect. Then, the mapping fit value is combined with the adjustment to achieve the best mapping effect.
[0041] In some embodiments, during S2, when transmitting the location and category information of people and objects in the physical scene to the MySQL database, the distance from the detected target to the camera is first obtained. In YOLOv5, distance measurement requires first obtaining the pixel coordinates of the target object and its actual size information in the real world. The method for obtaining this information is as follows:
[0042] Obtain the pixel coordinates of the target in the image;
[0043] Obtain the size of the target in the real world: Obtain the size of the target in the real world manually or with other general measuring tools and equipment, using the length of the object when it is at rest as the target size;
[0044] Obtain camera parameters: including the camera's focal length, optical center, and horizontal field of view.
[0045] Distance calculation using the principle of triangle similarity: Assuming the actual size of the target is known, a triangle can be constructed using the target's pixel coordinates and camera parameters to calculate the distance from the target to the camera. The specific formula is as follows:
[0046]
[0047] Where D is the detection distance, F represents the size of the target being detected, W represents the camera focal length, and h is the height of the detection box.
[0048] In some embodiments, in S3, by establishing a one-to-one correspondence between the twin system coordinate system and the physical world coordinate system, the detection distance is mapped to the front-back distance in the twin system, and the mapping formula is:
[0049] y = kx + b;
[0050] Where y is the vertical distance in the twin classroom, x is the detection distance, k is the slope, and b is a constant; the horizontal mapping is a linear mapping in one variable through the x-coordinate of the center coordinate and the horizontal distance of the twin classroom. Due to the perspective effect, the horizontal mapping, compared with the vertical mapping, needs to determine the slope k and constant b through the y-coordinate of the center coordinate, which is also a linear mapping in one variable.
[0051] A video-based virtual-real scene mapping system is used to implement the video-based virtual-real scene mapping method described above.
[0052] The video-based virtual-real scene mapping system includes:
[0053] The video monitoring module is used to acquire real-time video information of the physics classroom through camera sensors;
[0054] The data transmission module is used to transmit video data from video surveillance to the streaming media server via the UDP network transmission protocol, and to realize bidirectional data transmission between the client and the server via Socket communication.
[0055] The data processing and storage module is used to process the video data from the streaming media server using the YOLOv5 object detection algorithm, obtain the category information and location data of people and objects in the classroom, and store the data in the MySQL database.
[0056] The data mapping module is used to map the location data of objects in the MySQL database to the corresponding twin scene.
[0057] A computer-readable storage medium stores computer instructions, which, when read by a computer, execute the aforementioned video-based virtual-real scene mapping method.
[0058] The specific technical concept of this invention is as follows:
[0059] This invention employs the YOLOv5 target recognition and detection algorithm to process video, extracting the positional data of people and objects in the scene. The processed data is then uploaded to a MySQL database, and the client and server transmit relevant data using the UDP and TCP data network transmission protocols. Based on the stored data, a 3D mapping of positions is achieved. By leveraging the one-to-one correspondence between camera coordinates and twin system coordinates, a near real-time method is implemented to map the positions of people and objects in the physical scene to their corresponding positions in the twin scene.
[0060] This invention comprises four main functional modules: a video monitoring module, a data transmission module, a data processing and storage module, and a data mapping module. These four modules process video sequentially to achieve a video-based method for mapping virtual and real scenes.
[0061] Video monitoring module: Using a monocular camera, it can achieve 1080P pixel accuracy, and the real-time frame rate can reach 30FPS, which can basically meet the recognition needs of the location of people and objects in the environment.
[0062] Data transmission module: This module mainly handles two transmission scenarios: 1. Transmission of classroom surveillance video to the server. Considering transmission speed, the UDP network transmission protocol is used. 2. Transmission of information between Python and Unity. Considering security issues, the TCP / IP network transmission protocol is used.
[0063] Data processing and storage module: This module primarily handles the physical location information of objects. It identifies and processes data such as the distance of the detected target from the monocular camera and the center pixel coordinates of the detected target in the image, and then stores the data in the database. First, it obtains the distance from the detected target to the camera. In YOLOv5, distance measurement requires obtaining the pixel coordinates of the target object and its actual size in the real world.
[0064] Method of obtaining:
[0065] Obtain the pixel coordinates of the target in the image.
[0066] Obtaining the size of a target in the real world: The size of a target in the real world can be obtained manually or with other general measuring tools and equipment. Generally, the length of the object when it is at rest is used as the target size.
[0067] Obtain camera parameters: including camera focal length, optical center, horizontal field of view, etc.
[0068] Distance is calculated using the principle of triangle similarity: Assuming the actual size of the target is known, a triangle can be constructed using the target's pixel coordinates and camera parameters, thus allowing the calculation of the distance from the target to the camera. See the formula below for details.
[0069] Formula 1: Parameter Formula
[0070] In the formula: D represents the detection distance, F represents the size of the target being detected, W represents the camera focal length, and h represents the height of the detection frame.
[0071] It is important to note that in actual operation, the scene within the camera's field of view must be on the same plane. Also, since detection is easily affected by light, the calculation results may have errors, so try to ensure that the detection environment is bright and clear.
[0072] The data mapping module primarily establishes a one-to-one correspondence between the twin system's coordinate system and the physical world's coordinate system. It maps the detection distance to the front-to-back distance within the twin system, where y is the longitudinal distance in the twin classroom, x is the detection distance, k is the slope, and b is a constant. Lateral mapping uses a linear mapping from the center's x-coordinate to the twin classroom's lateral distance. However, due to perspective effects, lateral mapping, compared to longitudinal mapping, requires determining the slope k and constant b using the center's y-coordinate; this is also a linear mapping.
[0073] y = kx + b; Formula 2: Mapping Formula
[0074] The real-time video processing workflow is as follows: First, real-time video is uploaded to a streaming media server. YOLOv5 then processes the uploaded video for recognition. Based on the previously trained model and the set confidence threshold, objects in the image whose bounding boxes reach the threshold undergo further recognition processing. The recognized data is then loaded into a MySQL database. Before setting the YOLO confidence parameters, classroom images are used for testing. Recall, precision, false positive rate, and average false positive rate are considered comprehensively to determine the optimal confidence value for different classroom environments, achieving the best recognition results. Finally, the mapping fit is combined with further adjustments to achieve the best mapping effect.
[0075] Mapping effect technical indicators
[0076] Mapping Fit Accuracy: A metric for mapping precision, calculated as the ratio of the difference between the actual coordinates of the physical scene in the three axes and their corresponding coordinates in the twin system. The closer to 0, the higher the mapping accuracy and the better the mapping effect. See formulas 3, 4, and 5.
[0077] Formula 3: Formula for calculating the fitting accuracy in the X-axis direction:
[0078]
[0079] Formula 4: Formula for calculating the fitting accuracy in the Y-axis direction:
[0080]
[0081] Formula 5: Formula for calculating fitting accuracy in the Z-axis direction:
[0082]
[0083] Where α, β, and γ represent the scaling ratios of the three axes when building a twin of x, y, and z.
[0084] In summary, the specific process of the video-based virtual-real scene mapping method is as follows:
[0085] Step 1: Deploy sensors (monocular cameras) on the site to acquire real-time information about the site.
[0086] The second step is to meticulously build the corresponding twin scene of the physical environment, realize the detailed restoration of the scene, and design the display of the scene's data panel.
[0087] Step 3: Train the YOLOv5 video monitoring model to achieve data recognition and processing, such as the identification of object categories in the environment, the distance of the detected target from the monocular camera, and the center pixel coordinates of the detected target in the image.
[0088] Step 4: Create a MySQL data table based on the information to be uploaded. The table should contain information such as: the object's category, center pixel coordinates, and the physical world coordinates obtained by calculating the pixel coordinates.
[0089] Step 5: Transmit the real-time environmental status monitored by the sensors to the streaming media server via the UDP network transmission protocol.
[0090] Step 6: Call YOLOv5 for recognition processing, which should yield the relevant data required for database creation in Step 4.
[0091] Step 7: Use the TCP network transmission protocol to establish a connection between Python and Unity to transfer relevant data.
[0092] Step 8: Connect Unity to the MySQL database, calculate and convert the physical world coordinates in the database into coordinates in the corresponding twin coordinate system using the C# scripting language, encapsulate relevant information such as object categories into objects, and map them into the twin scene.
[0093] Step 9: Calculate the relevant mapping metrics and check if they are within the tolerance range. If they are not within the tolerance range, adjust the relevant YOLOv5 recognition metrics and return to Step 3 for remapping.
[0094] In a specific embodiment of video-based virtual-real scene mapping in the classroom:
[0095] The scene mapping can be mainly divided into four functional modules: video monitoring module, data transmission module, data processing and storage module, and data mapping module.
[0096] The video monitoring module acquires real-time video information of the physical classroom through camera sensors and stores the acquired data in the database. The purpose of the video monitoring module is to provide classroom environment data and real-time monitoring data of teacher and student activities for the classroom scene mapping.
[0097] The data transmission module transmits video data from the video surveillance system to the streaming media server via the UDP network protocol. It also utilizes Socket communication, allowing bidirectional data transmission between the client (Unity) and server (Unity), and enables the use of input / output streams for reading and writing data.
[0098] The data processing module takes the video data from the streaming media server and processes it using the YOLOv5 object detection algorithm to obtain the category information and location data of people and objects in the classroom, and then stores the data in the MySQL database.
[0099] The data mapping module maps the position data of objects in the MySQL database to the corresponding twin scene. The data stored in the MySQL database, analyzed and processed by the YOLOv5 object detection algorithm, includes the total number of detected targets, the center coordinates of each detected target, and the distance from each detected target to the camera. Classroom mapping primarily connects the physical classroom and the twin classroom using a linear equation (e.g., ...). Figure 1 (See Formula 2) Perform horizontal and vertical mapping and then further generate the model corresponding to the detection target.
[0100] This virtual-real scene mapping of classrooms based on classroom surveillance video can reflect the real-time situation of teachers and students as well as the environmental conditions in the classroom, and can improve the ability to respond to emergencies in public places.
[0101] In a specific embodiment of video-based virtual-real scene mapping in a conference room:
[0102] Similar to the specific implementation of video-based virtual-real scene mapping in classrooms, virtual-real scene mapping in conference halls can map members and objects in the conference hall to a twin conference hall scene in real time, and can display the progress of the meeting and the environmental conditions of the venue in real time.
[0103] In a specific embodiment of video-based virtual-real scene mapping in shopping malls:
[0104] Similar to the specific implementation of video-based virtual-real scene mapping in classrooms, this method can map people and facilities in a shopping mall into a virtual shopping mall scene in real time. It can display the flow and density of people in the mall at different times in real time, which can facilitate the mall's resource allocation decisions and improve its ability to respond to emergencies.
[0105] The application scenarios of this invention are not limited to specific scenarios. Many public places, such as classrooms, shopping malls, and conference halls, can use this invention to realize the mapping of physical real scenes to twin scenes, that is, the solution of this invention has reusability.
[0106] The embodiments described above are for illustrative purposes only and are not intended to limit the invention. Therefore, any changes in numerical values or substitutions of equivalent elements should still fall within the scope of this invention.
[0107] The above detailed description will enable those skilled in the art to understand that the present invention can indeed achieve the aforementioned objectives and has complied with the provisions of the Patent Law.
[0108] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.
[0109] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.
[0110] The basic concepts have been described above. Obviously, for those skilled in the art who have read this application, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore, such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.
[0111] Furthermore, this application uses specific terms to describe its embodiments. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.
[0112] Furthermore, those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Therefore, aspects of this application can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. All of the above hardware or software can be referred to as a “unit,” “module,” or “system.” Furthermore, aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.
[0113] The computer program code required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, and Python; general programming languages such as C; Visual Basic, Fortran2103, Perl, COBOL2102, PHP, and ABAP; dynamic programming languages such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).
[0114] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this application are not intended to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention have been discussed in the foregoing disclosure by way of various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a purely software solution, such as an installation on an existing server or mobile device.
[0115] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this approach of the present application should not be construed as reflecting an intention that the claimed subject matter requires more features than expressly recited in each claim. Rather, the subject of the invention should possess fewer features than in any single embodiment described above.
Claims
1. A video-based virtual-real scene mapping method, characterized in that, Includes the following steps: S1. Acquire the monitoring video stream in real time and upload it to the streaming media server via UDP protocol; S2. The video stream is processed using a YOLOv5 object detection model deployed on a server to obtain the category information, pixel coordinates, and detection box information of dynamic objects in the physical scene; Among them, distance measurement methods for dynamic targets include: S2.
1. Obtain the pixel coordinates of the target in the image and the height h of the detection box; S2.
2. Obtain the actual length F of the target when it is stationary in the real world; S2.
3. Obtain the camera's focal length W parameter; S2.
4. Based on the principle of triangle similarity, calculate the distance D from the target to the camera according to the formula D=(F×W) / h; S3. Transmit and store the target category, pixel coordinates and calculated distance D obtained in S2 to the MySQL database via TCP protocol; S4. Connect to the MySQL database using the Unity3D engine and read the target data; S5. Establish the mapping relationship between the physical coordinate system and the twin system coordinate system, mapping the target to the twin scene, where: S5.
1. The vertical mapping adopts a linear function y = k1x + b1, where y is the vertical coordinate of the twin scene and x is the distance D calculated in step S2.4; S5.
2. The horizontal mapping adopts a linear function based on perspective compensation, whose slope k2 and intercept b2 are dynamically determined according to the vertical pixel coordinates of the target in the image to achieve the horizontal coordinate mapping. S6. Real-time calculation of mapping fitting accuracy for closed-loop verification, wherein the mapping fitting accuracy is the ratio of the difference between the physical world coordinates and the twin system coordinates adjusted by the scaling ratio to the physical world coordinates. If the fitting accuracy exceeds the preset tolerance range, the feedback is sent to step S2 to adaptively adjust the confidence threshold or mapping parameters of the YOLOv5 model.
2. A video-based virtual-real scene mapping system, characterized in that, The system for implementing the method of claim 1 includes: The video capture and transmission module is used to capture real-time video of the physical scene through a camera and upload it to the streaming media server via the UDP protocol; The data processing and ranging module, deployed on the server, is used to run the YOLOv5 model to identify dynamic targets and calculate the target distance according to the method of steps S2.1 to S2.4 in claim 1; The data storage and relay module is used to receive and store the target category, coordinates, and distance data to a MySQL database via the TCP protocol. The virtual-real mapping and rendering module, built on the Unity3D engine, is used to read data from the database and generate and render the corresponding dynamic target model in the twin scene according to the mapping relationship in steps S5.1 and S5.2 of claim 1. The closed-loop verification and optimization module is used to perform the fitting accuracy calculation and parameter adaptive adjustment in step S6 of claim 1, and to feed back the adjustment instructions to the data processing and ranging module and / or the virtual-real mapping and rendering module.
3. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the video-based virtual-real scene mapping method as described in claim 1.