Position determination method, device, electronic device and storage medium

Abstract

The present invention relates to the field of computer vision technology, and discloses a posture determination method, device, electronic device and storage medium. The present invention provides a posture determination method, including: extracting a target frame image from target video data, the target video data is obtained by collecting images of a target scene by a physical camera; constructing a three-dimensional model of the target scene; extracting multiple feature point data in the three-dimensional model, and constructing a simulated image based on the multiple feature point data; based on the matching result between the simulated image and the target frame image, determining the posture data when the physical camera collects the target frame image, the target frame image includes multiple feature point data. It can effectively reduce the cost of manual measurement and achieve the purpose of improving the efficiency of posture data determination.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, electronic device and storage medium for determining a position and posture. Background Art

[0002] Digital twin is a technology that matches the digital representation of a physical object or system with a virtual model corresponding to its actual state, thereby achieving real-time monitoring, simulation and analysis. In digital twin, various different data and materials need to be integrated. In order to integrate the video materials collected by the camera into the corresponding three-dimensional model, the real world coordinate system needs to be accurately matched and matched with the virtual world coordinate system.

[0003] In the related art, in order to determine the position of the camera when collecting video data, relevant personnel need to use the method of field measurement, using a total station (universal measuring instrument) to determine it. However, when the target scene to be three-dimensionally modeled is large, using this method to determine the position of the camera requires a lot of labor costs and is inefficient. Summary of the invention

[0004] In view of this, the present invention provides a posture determination method, device, electronic device and storage medium to solve the problem of low efficiency and high cost of camera posture determination.

[0005] In a first aspect, the present invention provides a method for determining a posture, the method comprising:

[0006] Extracting a target frame image from target video data, where the target video data is obtained by collecting images of a target scene through a physical camera;

[0007] Construct a 3D model of the target scene;

[0008] Extracting multiple feature point data from the three-dimensional model, and constructing a simulated image based on the multiple feature point data;

[0009] Based on the matching result between the simulation image and the target frame image, the pose data of the physical camera when acquiring the target frame image is determined, and the target frame image includes a plurality of feature point data.

[0010] Beneficial effect: The posture determination method provided in this embodiment can effectively reduce the cost of manual measurement, and determine the posture data when the physical camera captures the target frame image based on the matching result between the simulated image in the virtual world coordinate system and the target frame image in the real world coordinate system, which can make the posture data determination method more efficient and convenient, thereby achieving the purpose of improving the efficiency of posture data determination.

[0011] In an optional implementation, the simulated image is obtained based on initial pose data of a virtual camera, the virtual camera corresponds to a physical camera and is deployed on a three-dimensional model; based on a matching result between the simulated image and the target frame image, determining the pose data when the physical camera captures the target frame image includes:

[0012] The simulated image is superimposed on the target frame image, and the projection coordinates of each feature point data projected onto the target frame image are determined respectively;

[0013] Determine the matching result between the simulated image and the target frame image based on the coordinate difference between the point coordinates of each feature point data on the target frame image and the corresponding projection coordinates;

[0014] Based on the matching results, the initial pose data is adjusted to obtain the pose data when the physical camera captures the target frame image.

[0015] Beneficial effect: Based on the matching results between the simulated image in the virtual world coordinate system and the target frame image in the real world coordinate system, it is possible to effectively evaluate whether the posture of the virtual camera when acquiring the simulated image is the same as the posture of the physical camera when acquiring the target frame image. Furthermore, when the simulated image is different from the target frame image, the initial posture data of the virtual camera can be adjusted in a targeted manner to achieve the purpose of quickly determining the posture data of the physical camera when acquiring the target frame image.

[0016] In an optional implementation, determining a matching result between the simulated image and the target frame image based on a coordinate difference between a point coordinate of each feature point data on the target frame image and a corresponding projection coordinate includes:

[0017] According to the coordinate difference between the point coordinates of each feature point data on the target frame image and the corresponding projection coordinates, variance processing is performed to obtain the position deviation value of multiple feature point data between the simulation image and the target frame image;

[0018] If the position deviation value is less than or equal to the preset threshold, it is determined that the matching results between the simulated image and the target frame image are the same;

[0019] If the position deviation value is greater than a preset threshold, it is determined that the matching result between the simulated image and the target frame image is different.

[0020] In an optional implementation, based on the matching result, the initial posture data is adjusted to obtain the posture data when the physical camera captures the target frame image, including:

[0021] If the matching results are the same, the initial pose data is used as the pose data when the physical camera captures the target frame image;

[0022] If the matching results are not the same, the initial pose data is adjusted until the matching results are the same, and the adjusted initial pose data is used as the pose data when the physical camera captures the target frame image.

[0023] In an optional implementation, the initial posture data is adjusted via a virtual reality display device.

[0024] Beneficial effect: It can improve the adjustment efficiency and make the adjusted initial posture data more in line with human senses.

[0025] In an optional implementation, extracting a target frame image from target video data includes:

[0026] Based on the result of target detection on the target video data, a target frame image is determined and extracted, and the target frame image is composed of at least one static object.

[0027] Beneficial effect: It can improve the stability of the target frame image and avoid the interference of invalid data, thereby improving the accuracy of determining the posture data.

[0028] In an optional implementation, the target frame image is pulled from a streaming media server in real time.

[0029] Beneficial effect: Key visual information can be obtained based on the video data collected in real time by the physical camera, so that the camera's position data can be better determined.

[0030] In a second aspect, the present invention provides a posture determination device, comprising:

[0031] A first extraction module is used to extract a target frame image from target video data, where the target video data is obtained by collecting images of a target scene through a physical camera;

[0032] An acquisition module is used to construct a three-dimensional model of the target scene;

[0033] A second extraction module is used to extract a plurality of feature point data in the three-dimensional model and construct a simulated image based on the plurality of feature point data;

[0034] The processing module is used to determine the posture data of the target frame image when the physical camera captures the target frame image based on the matching result between the simulated image and the target frame image, and the target frame image includes multiple feature point data.

[0035] In a third aspect, the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the posture determination method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0036] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the posture determination method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a structural schematic diagram of a total station according to the prior art;

[0039] Figure 2 is a flow chart of a method for determining a posture according to an embodiment of the present invention;

[0040] Figure 3 is a flow chart of another method for determining a posture according to an embodiment of the present invention;

[0041] Figure 4 is a structural block diagram of another apparatus for determining a posture according to an embodiment of the present invention;

[0042] Figure 5 is a schematic diagram of a structural framework of a streaming media server according to an embodiment of the present invention;

[0043] Figure 6 A schematic flow chart of another method for determining a posture according to an embodiment of the present invention;

[0044] Figure 7 is a structural block diagram of another apparatus for determining a posture according to an embodiment of the present invention;

[0045] Figure 8 It is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0047] In the related art, in order to determine the position of the camera when collecting video data, relevant personnel need to use field measurement methods, such as Figure 1 The total station (all-purpose measuring instrument) shown in the figure is used for determination. The total station adopts the principle of geometric measurement, and establishes the geometric relationship between the external known point and the point to be measured by observing the coordinates and azimuth of the external known point, as well as the measured angle and distance between the external known point and the point to be measured, and then determines the latitude, longitude and altitude information of the point to be measured through calculation and deduction.

[0048] Therefore, in order to accurately correspond and match the real world coordinate system and the virtual world coordinate system when the camera collects video data, relevant personnel need to continuously measure and debug through the total station to determine the position and posture of the camera collecting video data. However, when the scope of the target scene is large, if this method is still used to determine the position and posture of the camera, it will not only consume a lot of labor costs, but also be extremely inefficient.

[0049] In view of this, an embodiment of the present invention provides a posture determination method, which can not only reduce labor costs but also improve the efficiency of determining posture data.

[0050] According to an embodiment of the present invention, an embodiment of a posture determination method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] In this embodiment, a method for determining a posture is provided, which can be used in electronic devices, including tablets, computers, etc. Figure 2 is a flow chart of a method for determining a posture according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0052] Step S201, extracting a target frame image from target video data.

[0053] In the implementation manner of the present invention, the target video data is obtained by capturing images of the target scene through a physical camera.

[0054] Step S202: construct a three-dimensional model of the target scene.

[0055] In the embodiment of the present invention, in order to more clearly understand and communicate the characteristics, structure and layout of the target scene, a three-dimensional model of the target scene is constructed. The target scene is a real scene in the physical world, for example, the target scene may include any of the following types of scenes: architectural scene, urban scene, industrial scene, natural scene, etc., which can be set according to actual needs and are not limited in the present invention.

[0056] In some examples, a three-dimensional model of the target scene can be constructed by oblique photography technology. Specifically, a drone is first used to collect photos of the target scene according to a preset cruise route. Since the position, posture and field of view of the drone are known, based on the aerial triangulation technology, the coordinates and elevations of multiple feature points of the target scene can be determined, and then mapped to the three-dimensional model surface to obtain a three-dimensional model of the target scene. Since aerial triangulation technology is relatively mature, it will not be described in detail here. Preferably, a drone with centimeter-level positioning accuracy can be used for photo collection, which helps to improve the collection progress, thereby ensuring that the obtained three-dimensional model is more in line with the target scene.

[0057] Step S203: extract multiple feature point data from the three-dimensional model, and construct a simulated image based on the multiple feature point data.

[0058] In an embodiment of the present invention, multiple feature point data are extracted from the three-dimensional model and projected onto the image plane coordinate system. Through image rendering, a simulated image of the multiple feature point data in the two-dimensional coordinate system is obtained, so as to estimate the pose data of the physical camera based on the mapping relationship between the image plane coordinates and the world coordinate system.

[0059] In one example, the feature point data may be any of the following types of feature points in the target scene: for example, feature points on static objects, feature points with obvious colors, or feature points on the edges and corners of an object surface. The feature point data may be extracted manually or by target detection, which is not limited in the present invention.

[0060] Step S204, based on the matching result between the simulation image and the target frame image, determine the posture data when the physical camera captures the target frame image.

[0061] In an embodiment of the present invention, the target frame image includes a plurality of feature point data. In order to improve the efficiency of posture determination, a target frame image including a plurality of feature point data is extracted from the target video data. Based on the matching result between the simulation image and the target frame image, the position deviation between the target frame image and the simulation image can be clarified, and then based on the position deviation between the two, the posture data of the physical camera during the simulation image acquisition is estimated through a preset algorithm, which can effectively reduce labor costs, thereby achieving the purpose of improving the efficiency of posture determination. For example: the preset algorithm can be a random sample consensus algorithm (RANSAC) or PnP (Perspective-n-Point, an algorithm for calculating camera posture).

[0062] The posture determination method provided in this embodiment can effectively reduce the cost of manual measurement, and determine the posture data when the physical camera captures the target frame image based on the matching result between the simulated image in the virtual world coordinate system and the target frame image in the real world coordinate system, which can make the method of determining the posture data more efficient and convenient, thereby achieving the purpose of improving the efficiency of posture data determination.

[0063] In an optional embodiment, in order to facilitate the construction of a simulated image, a virtual camera corresponding to the physical camera is deployed in the three-dimensional model, and the pose data of the simulated image collected by the virtual camera is used to restore the pose of the target frame image collected by the physical camera, thereby achieving the purpose of improving the efficiency of determining the pose data.

[0064] Among them, the expression of the mapping relationship between the image plane coordinates and the world coordinate system is as follows:

[0065]

[0066] in, Used to represent the internal parameters of the camera; Used to represent the external parameters of the camera; u and v represent the pixel coordinates of the feature point data in the image plane coordinate system, X, Y, and Z represent the three-dimensional coordinates of the feature point data restored to the world coordinate system, R represents the rotation matrix of the virtual camera, T represents the translation vector of the virtual camera, and f x and f y is the focal length of the simulated image, u 0 and v 0 is the coordinate of the center point of the simulated image. The internal parameters of the virtual camera are the same as those of the example camera, which are known data provided by the manufacturer of the physical camera. The external parameters of the virtual camera are the pose data of the physical camera that needs to be determined in the end. In order to ensure that the simulated image can be constructed successfully, the initial pose parameters of the virtual camera are pre-configured so that the adjustment direction can be clearly determined during subsequent adjustments to improve the adjustment efficiency.

[0067] In this embodiment, a method for determining a posture is provided, which can be used for the above-mentioned electronic devices, including: tablets, computers, etc. Figure 3 is a flow chart of a method for determining a posture according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0068] Step S301, extracting a target frame image from the target video data. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0069] Step S302: construct a 3D model of the target scene. Figure 2Step S202 of the illustrated embodiment will not be described in detail here.

[0070] Step S303: extract multiple feature point data from the three-dimensional model, and construct a simulated image based on the multiple feature point data. Figure 2 Step S203 of the illustrated embodiment will not be described in detail here.

[0071] Step S304, based on the matching result between the simulation image and the target frame image, determine the posture data when the physical camera captures the target frame image.

[0072] Specifically, the above step S304 includes:

[0073] Step S3041 , superimposing the simulated image on the target frame image, and determining the projection coordinates of each feature point data projected onto the target frame image.

[0074] In an embodiment of the present invention, in order to determine whether the positions of multiple feature point data on the target frame image are different from their positions on the simulated image, the simulated image is superimposed on the target frame image, and then the projection coordinates of each feature point data projected onto the target frame image are determined respectively, so that the position deviation between the projection positions of multiple feature point data on the target frame image and their actual positions on the target frame image can be more intuitively and quickly determined.

[0075] Step S3042, determining the matching result between the simulation image and the target frame image based on the coordinate difference between the point coordinates of each feature point data on the target frame image and the corresponding projection coordinates.

[0076] In an embodiment of the present invention, in order to determine whether the simulated image obtained by the virtual camera according to the initial posture data is the same as the posture data when the physical camera captures the target frame image, the coordinate difference between the point coordinates of each feature point data on the target frame image and the corresponding projection coordinates is determined respectively, and then the matching between the simulated image and the target frame image is detected by the coordinate difference corresponding to each feature point data to obtain a matching result, so that the adjustment direction of the initial posture data can be clarified according to the matching result, which helps to improve the adjustment efficiency.

[0077] In some optional implementations, the above step S3042 includes:

[0078] Step a1, performing variance processing according to the coordinate difference between the point coordinates of each feature point data on the target frame image and the corresponding projection coordinates, to obtain the position deviation values ​​of multiple feature point data between the simulation image and the target frame image.

[0079] Step a2: If the position deviation value is less than or equal to the preset threshold, it is determined that the matching result between the simulated image and the target frame image is the same.

[0080] Step a3: If the position deviation value is greater than a preset threshold, it is determined that the matching result between the simulated image and the target frame image is different.

[0081] Specifically, in order to improve matching efficiency and save computing costs, after determining the coordinate difference corresponding to each feature point data, variance processing is performed on all coordinate differences to determine the overall position deviation of multiple feature point data, thereby obtaining the position deviation value of multiple feature point data between the simulated image and the target frame image.

[0082] The preset threshold value can be understood as the maximum position deviation value that is reasonable to consider the position difference. If the position deviation value is less than or equal to the preset threshold value, it is reasonable to represent that the position difference of multiple feature point data between the simulated image and the target frame image, and then it can be determined that the matching result between the simulated image and the target frame image is the same. If the position deviation value is greater than the preset threshold value, it is unreasonable to represent that the position difference of multiple feature point data between the simulated image and the target frame image, and then it can be determined that the matching result between the simulated image and the target frame image is different.

[0083] Step S3043, based on the matching result, adjust the initial posture data to obtain the posture data when the physical camera captures the target frame image.

[0084] In the embodiment of the present invention, since the virtual camera is a camera in the virtual world corresponding to the physical camera, the initial posture data can be used as the estimated posture data when the physical camera collects the target frame image. Based on the matching results, it can be determined whether the posture of the virtual camera when collecting the simulated image is the same as the posture of the physical camera when collecting the target frame image, and then it can be further determined whether the initial posture data needs to be adjusted in a targeted manner, thereby helping to improve the efficiency of determining the posture data when the physical camera collects the target frame image.

[0085] In some optional implementations, the above step S3043 includes:

[0086] Step b1: If the matching results are the same, the initial posture data is used as the posture data when the physical camera captures the target frame image.

[0087] Step b2: If the matching results are not the same, the initial posture data is adjusted until the matching results are the same, and the adjusted initial posture data is used as the posture data when the physical camera captures the target frame image.

[0088] Specifically, if the matching results are the same, it means that the posture of the virtual camera when capturing the simulated image is the same as the posture of the physical camera when capturing the target frame image. There is no need to adjust the initial posture data, and the initial posture data can be directly used as the posture data when the physical camera captures the target frame image.

[0089] If the matching result is different, it means that the pose of the virtual camera when acquiring the simulation image is different from the pose of the physical camera when acquiring the target frame image, and it is necessary to determine the pose data of the physical camera when acquiring the target frame image by adjusting the initial pose data. Therefore, it is necessary to repeat the steps S302 to S3043 until the matching result is the same, and then use the adjusted initial pose data as the pose data of the physical camera when acquiring the target frame image.

[0090] In some optional implementation scenarios, the initial posture data can be adjusted by a virtual reality display device. Among them, posture data refers to position data and posture data, the position data includes the three-dimensional spatial position (x, y, z) of the physical camera in the world coordinate system, and the posture data includes roll, yaw and pitch. That is, the posture data is composed of data in 6 dimensions. Therefore, in order to improve the adjustment efficiency and make the adjusted initial posture data more in line with human senses, a virtual reality display device is used for auxiliary adjustment. In one example, a virtual reality (VR) display device is a wearable device (e.g., a head-mounted display device) as an example. In the process of adjusting the initial posture data using a VR display device, the combined image of the simulated image superimposed on the target frame image can be sent to the VR display device for display, and the user of the VR display device adjusts the initial posture data by adjusting the position and viewing angle of the head, which can not only improve the operating efficiency, but also greatly reduce the professionalism of the relevant personnel. In another example, the VR display device may also be an electronic display screen, and the user of the VR display device can adjust the initial posture data by adjusting the left and right handles, thereby improving operational efficiency and reducing the professionalism of related personnel.

[0091] The posture determination method provided in this embodiment is based on the matching results between the simulated image in the virtual world coordinate system and the target frame image in the real world coordinate system. It can effectively evaluate whether the posture of the virtual camera when capturing the simulated image is the same as the posture of the physical camera when capturing the target frame image. Furthermore, when the simulated image is different from the target frame image, the initial posture data of the virtual camera can be adjusted in a targeted manner to achieve the purpose of quickly determining the posture data of the physical camera when capturing the target frame image.

[0092] In this embodiment, a method for determining a posture is provided, which can be used for the above-mentioned electronic devices, including: tablets, computers, etc. Figure 4 is a flow chart of a method for determining a posture according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0093] Step S401: extracting a target frame image from target video data.

[0094] Specifically, the above step S401 includes:

[0095] Step S4011, based on the result of target detection on the target video data, determine and extract the target frame image.

[0096] In an embodiment of the present invention, the target frame image is composed of at least one static object. In order to improve the accuracy of determining the pose data and avoid interference from dynamic objects, target detection is performed on the target video data to identify the frame image containing only the static object in the target video data, so as to extract it as the target frame image.

[0097] In one example, based on the results of target detection, there may be multiple frames of images containing only static objects in the target video data. Therefore, a frame image can be randomly selected from the multiple frames of images containing only static objects through random sampling and extracted as the target frame image.

[0098] Step S402: construct a three-dimensional model of the target scene.

[0099] Step S403: extract multiple feature point data from the three-dimensional model, and construct a simulated image based on the multiple feature point data.

[0100] Step S404, based on the matching result between the simulation image and the target frame image, determine the posture data when the physical camera captures the target frame image.

[0101] The posture determination method provided in this embodiment can improve the stability of the target frame image and avoid interference from invalid data, thereby improving the accuracy of determining the posture data.

[0102] In some optional implementations, the target frame image is pulled from the streaming media server in real time, and then key visual information can be obtained based on the video data collected by the physical camera in real time, so as to better determine the camera's position data. In one example, the streaming media server can be a local GBS (Game Broadcasting Server, a server for receiving, encoding and transmitting real-time audio and video streams of game content to users). The camera can pull the image from the GBS via the Real Time Streaming Protocol (RTSP). In some optional implementation scenarios, the structural framework of the streaming media server can be as follows: Figure 5 As shown in Figure 1, the IoT system includes a device terminal layer, a national standard platform layer, a simple global business service layer, and an application layer. Among them, the device terminal layer includes various IoT terminal devices, such as network cameras, network video recorder devices, etc. These devices can sense the environment, collect data, and communicate with other devices. The device terminal layer is mainly used to collect environmental data and transmit it to the national standard platform for processing and analysis. The national standard platform layer is used to receive data transmitted by the device terminal and process, analyze and store it. The simple global business service layer is used to provide various business applications and services, as well as integrate the data and functions provided by the national standard platform, and provide users with personalized and intelligent services through technologies such as cloud computing, big data analysis and artificial intelligence. The application layer is the top-level application interface of the IoT system for direct use by users. The application layer can be a mobile application, a web application, a desktop application, etc. Users can receive services from the simple global business service layer through the application layer and interact with the IoT system. The design of the application layer should be simple and easy to use, and provide rich functions and personalized setting options.

[0103] In some optional implementation scenarios, such as Figure 6 As shown, the process of determining the position and posture of the virtual camera can be as follows:

[0104] Step S601: pull the target video data for real-time acquisition of the target scene through the streaming media server, and extract the target frame image.

[0105] Step S602: construct a three-dimensional model of the target scene.

[0106] Step S603: extract multiple feature point data from the three-dimensional model, and construct a simulated image based on the multiple feature point data and the initial position data of the virtual camera.

[0107] Step S604, superimposing the simulated image on the target frame image, and determining the matching result between the simulated image and the target frame image based on the coordinate difference between the point coordinates of each feature point data on the target frame image and the corresponding projection coordinates.

[0108] Step S605: Based on the matching result, the initial posture data is adjusted through the virtual reality display device to obtain the posture data when the physical camera captures the target frame image.

[0109] In the posture determination method provided by the present invention, the posture data of the physical camera is determined by virtual measurement, which can greatly save labor costs. In the posture judgment and adjustment steps, a virtual reality display device is used for adjustment, so that the adjusted initial posture data is more in line with human intuition, which is conducive to improving operation efficiency.

[0110] In this embodiment, a posture determination device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0111] This embodiment provides a posture determination device, such as Figure 7 As shown, including:

[0112] The first extraction module 701 is used to extract a target frame image from target video data, where the target video data is obtained by collecting images of a target scene through a physical camera;

[0113] An acquisition module 702 is used to construct a three-dimensional model of a target scene;

[0114] A second extraction module 703 is used to extract a plurality of feature point data in the three-dimensional model and construct a simulated image based on the plurality of feature point data;

[0115] The processing module 704 is used to determine the posture data of the target frame image when the physical camera captures the target frame image based on the matching result between the simulated image and the target frame image, and the target frame image includes multiple feature point data.

[0116] In some optional embodiments, the simulated image is obtained based on the initial pose data of a virtual camera, which corresponds to a physical camera and is deployed on a three-dimensional model; the processing module 704 includes: a first execution unit, used to superimpose the simulated image on the target frame image, and respectively determine the projection coordinates of each feature point data projected onto the target frame image; a matching unit, used to determine the matching result between the simulated image and the target frame image based on the coordinate difference between the point coordinates of each feature point data on the target frame image and the corresponding projection coordinates; an adjustment unit, used to adjust the initial pose data based on the matching result, and obtain the pose data when the physical camera captures the target frame image.

[0117] In some optional embodiments, the matching unit includes: a first processing unit, used to perform variance processing based on the coordinate difference between the point coordinates of each feature point data on the target frame image and the corresponding projection coordinates, so as to obtain position deviation values ​​of multiple feature point data between the simulated image and the target frame image; a first determination unit, used to determine that the matching results between the simulated image and the target frame image are the same if the position deviation value is less than or equal to a preset threshold; and a second determination unit, used to determine that the matching results between the simulated image and the target frame image are different if the position deviation value is greater than a preset threshold.

[0118] In some optional embodiments, the adjustment unit includes: a second execution unit, which is used to use the initial pose data as the pose data when the physical camera captures the target frame image if the matching results are the same; and a third execution unit, which is used to adjust the initial pose data until the matching results are the same if the matching results are not the same, and use the adjusted initial pose data as the pose data when the physical camera captures the target frame image.

[0119] In some optional implementations, the initial posture data is adjusted via a virtual reality display device.

[0120] In some optional implementations, the first extraction module 701 includes: an image extraction module, configured to determine and extract a target frame image based on a result of target detection on target video data, wherein the target frame image is composed of at least one static object.

[0121] In some optional implementations, the target frame image is pulled from a streaming media server in real time.

[0122] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0123] The posture determination device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0124] The embodiment of the present invention also provides an electronic device having the above Figure 7 The posture determination device shown.

[0125] See also Figure 8 , Figure 8 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, such as Figure 8As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0126] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0127] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0128] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0129] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0130] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0131] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0132] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0133] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for determining a posture, characterized in that: The method comprises: Extracting a target frame image from target video data, wherein the target video data is obtained by performing image acquisition on a target scene by a physical camera, and the target frame image is composed of at least one static object; Constructing a three-dimensional model of the target scene; Extracting a plurality of feature point data from the three-dimensional model, and constructing a simulated image based on the plurality of feature point data; Based on the matching result between the simulation image and the target frame image, the posture data when the physical camera collects the target frame image is determined, the target frame image includes the multiple feature point data, and the matching result is used to determine the adjustment direction of the initial posture data, so as to adjust the initial posture data using a virtual reality display device; wherein, the virtual reality display device is a wearable device or an electronic display screen. If the virtual reality display device is a wearable device, the process of adjusting the initial posture data using the virtual reality display device includes: sending a combined image in which the simulation image is superimposed on the target frame image to the virtual reality display device for display, and the user of the virtual reality display device adjusts the initial posture data by adjusting the position and viewing angle of the head; if the virtual reality display device is an electronic display screen, the user of the virtual reality display device adjusts the initial posture data by adjusting the left and right handles.

2. The method according to claim 1, characterized in that The simulated image is obtained based on initial posture data of a virtual camera, and the virtual camera corresponds to the physical camera and is deployed on the three-dimensional model; The determining, based on the matching result between the simulation image and the target frame image, the pose data when the physical camera acquires the target frame image comprises: The simulated image is superimposed on the target frame image, and the projection coordinates of each feature point data projected onto the target frame image are determined respectively; Determine a matching result between the simulated image and the target frame image based on a coordinate difference between a point coordinate of each feature point data on the target frame image and a corresponding projection coordinate; Based on the matching result, the initial posture data is adjusted to obtain the posture data when the physical camera captures the target frame image.

3. The method according to claim 2, characterized in that The determining of the matching result between the simulated image and the target frame image based on the coordinate difference between the point coordinates of each feature point data on the target frame image and the corresponding projection coordinates includes: Performing variance processing according to the coordinate difference between the point coordinates of each feature point data on the target frame image and the corresponding projection coordinates to obtain position deviation values ​​of the plurality of feature point data between the simulation image and the target frame image; If the position deviation value is less than or equal to a preset threshold, determining that the matching result between the simulated image and the target frame image is the same; If the position deviation value is greater than the preset threshold, it is determined that the matching result between the simulation image and the target frame image is different.

4. The method according to claim 3, characterized in that The adjusting the initial posture data based on the matching result to obtain the posture data when the physical camera captures the target frame image includes: If the matching results are the same, the initial posture data is used as the posture data when the physical camera captures the target frame image; If the matching results are not the same, the initial posture data are adjusted until the matching results are the same, and the adjusted initial posture data are used as the posture data when the physical camera captures the target frame image.

5. The method according to claim 1, characterized in that The step of extracting a target frame image from target video data comprises: Based on the result of target detection on the target video data, the target frame image is determined and extracted.

6. The method according to claim 5, characterized in that The target frame image is obtained by pulling it from the streaming media server in real time.

7. A posture determination device, characterized in that: The device comprises: An acquisition module is used to construct a three-dimensional model of the target scene; A first extraction module, used to extract a plurality of feature point data in the three-dimensional model, and construct a simulated image based on the plurality of feature point data; A second extraction module is used to extract a target frame image from the target video data, wherein the target video data is obtained by performing image acquisition on the target scene by a physical camera, the target frame image includes the plurality of feature point data, and the target frame image is composed of at least one static object; A processing module is used to determine the posture data when the physical camera collects the target frame image based on the matching result between the simulated image and the target frame image, and the matching result is used to determine the adjustment direction of the initial posture data, so as to adjust the initial posture data using a virtual reality display device; wherein the virtual reality display device is a wearable device or an electronic display screen. If the virtual reality display device is a wearable device, the process of adjusting the initial posture data using the virtual reality display device includes: sending a combined image in which the simulated image is superimposed on the target frame image to the virtual reality display device for display, and the user of the virtual reality display device adjusts the initial posture data by adjusting the position and viewing angle of the head; if the virtual reality display device is an electronic display screen, the user of the virtual reality display device adjusts the initial posture data by adjusting the left and right handles.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the posture determination method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the posture determination method according to any one of claims 1 to 6.

Images