Image processing method, device, equipment and storage medium

Abstract

The embodiment of the present application provides an image processing method, device, equipment and storage medium, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and vehicle-mounted. The method includes: after obtaining the image to be processed shot by a real shooting device, the first position information of the position reference pattern in the image to be processed in the image coordinate system and the second position information in the world coordinate system are used to determine the posture information of the real shooting device, so that the shooting of the image to be processed and the acquisition of the posture information are synchronously aligned, so based on the internal reference information and the posture information, the target object image in the image to be processed is fused with the virtual background image, and when the target image is obtained, the accuracy of the picture rendering is guaranteed, thereby improving the presentation effect of the video obtained by virtual production. There is no need to use professional-level hardware to solve the problem of synchronization between the locator and the shot picture, which reduces the cost of virtual production and is conducive to the widespread application of virtual production.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] Virtual production technology introduces real objects into virtual scenes, building an interactive feedback information loop between the virtual world, the real world and real objects to enhance the sense of reality. This mixed reality technology between reality and virtuality has become a cutting-edge practical technology in the fields of display, experience, training, medical treatment, monument restoration, industrial maintenance, etc.

[0003] When performing virtual production, the relevant technology installs a locator on the shooting device to obtain the posture information of the shooting device, and uses the posture information to control the rendering of the virtual scene, and then superimposes the subject (people, objects, etc.) in the real scene captured by the shooting device onto the virtual scene.

[0004] However, the above method is prone to the situation where the time when the locator obtains the posture information is not synchronized with the time when the camera captures the picture. This will cause the virtual scene to not match the real subject, resulting in rendering errors in the picture, and then resulting in poor presentation effect of the video obtained by virtual production. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device and storage medium for improving the accuracy of picture rendering, thereby improving the presentation effect of the video obtained by virtual production.

[0006] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0007] Acquire an image to be processed captured by a real shooting device, wherein the image to be processed includes an image of a target object;

[0008] If at least one position reference pattern is detected in the image to be processed, respectively determining the at least one position parameter pattern, first position information of each pattern in the image coordinate system, and second position information of each pattern in the world coordinate system;

[0009] Determine target position information of the real shooting device based on each first position information, each second position information, and internal reference information of the real shooting device;

[0010] Based on the internal reference information and the target pose information, the target object image is fused with a virtual background image to obtain a target image.

[0011] On the one hand, an embodiment of the present application provides an image processing device, the device comprising:

[0012] An acquisition module, used to acquire an image to be processed taken by a real shooting device, wherein the image to be processed includes an image of a target object;

[0013] A detection module, configured to respectively determine the at least one position parameter pattern, first position information of each of the at least one position parameter pattern in the image coordinate system, and second position information of each of the at least one position parameter pattern in the world coordinate system if at least one position reference pattern is detected in the image to be processed;

[0014] A processing module, configured to determine target position information of the real shooting device based on each first position information, each second position information, and internal reference information of the real shooting device;

[0015] A fusion module is used to fuse the target object image with the virtual background image based on the internal reference information and the target posture information to obtain a target image.

[0016] Optionally, the detection module is further used for:

[0017] If no position reference pattern is detected in the image to be processed, the historical posture information corresponding to the previous image to be processed shot by the real shooting device is used as the target posture information.

[0018] Optionally, the detection module is specifically used to:

[0019] For the at least one position reference pattern, the following steps are performed respectively:

[0020] At least one feature point contained in a position reference pattern obtained by detection, the corresponding feature point numbers, and the first feature point coordinates of each feature point in the image coordinate system are used as the first position information of the position reference pattern.

[0021] Optionally, the detection module is specifically used to:

[0022] Based on the pattern number embedded in the one position reference pattern obtained by detection, query and obtain the at least one feature point, the corresponding feature point number, and the second feature point coordinates of each feature point in the world coordinate system;

[0023] The obtained characteristic point numbers and the corresponding second characteristic point coordinates are used as the second position information of the one position reference pattern.

[0024] Optionally, the processing module is specifically used for:

[0025] For the at least one position reference pattern, the following steps are performed respectively:

[0026] Based on the feature point number corresponding to each of the at least one feature point included in a position reference pattern, first feature point coordinates and second feature point coordinates corresponding to each of the at least one feature point are grouped to obtain a feature point coordinate pair set corresponding to the one position reference pattern, wherein each feature point coordinate pair in the feature point coordinate pair set includes a first feature point coordinate and a second feature point coordinate;

[0027] Based on the feature point coordinate pair sets corresponding to each of the at least one position reference patterns and the internal parameter information of the real shooting device, the target position and pose information of the real shooting device is determined.

[0028] Optionally, the image to be processed also includes a real background image;

[0029] Also includes screenshot module;

[0030] The screenshot module is specifically used for:

[0031] Based on the internal reference information and the target pose information, the target object image is fused with the virtual background image, and before obtaining the target image, third position information of the boundary line between the green screen area and the non-green screen area in the real background image in the world coordinate system is obtained;

[0032] Determine fourth position information of the boundary line in the image coordinate system based on the third position information, the internal reference information and the target pose information;

[0033] Removing the image of the green screen area from the image to be processed to obtain a preprocessed image;

[0034] Based on the fourth position information, the image of the non-green screen area outside the boundary line in the preprocessed image is removed to obtain the target object image.

[0035] Optionally, the fusion module is specifically used for:

[0036] Based on the internal reference information, determining the field of view angle of the real shooting device;

[0037] Based on the internal reference information, the target posture information, and the field of view angle, determining attribute information of a virtual shooting device in a virtual world through a rendering engine;

[0038] Determining target position information of a target imaging plane of the target object image in the virtual world based on the attribute information of the virtual shooting device and the reference position point in the target object image;

[0039] Based on the target position information of the target imaging surface, the target object image is fused with the virtual background image to obtain a target image.

[0040] Optionally, the fusion module is specifically used for:

[0041] Determining target size information of the target imaging plane based on the attribute information of the virtual shooting device and the target position information of the target imaging plane;

[0042] Based on the target position information and the target size information, the target object image is fused with a virtual background image to obtain a target image.

[0043] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned image processing method when executing the program.

[0044] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned image processing method.

[0045] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned image processing method.

[0046] In the embodiment of the present application, after obtaining the image to be processed shot by the real shooting device, the target posture information of the real shooting device is determined by the position reference pattern in the image to be processed, the first position information in the image coordinate system, and the second position information in the world coordinate system, so that the shooting of the image to be processed and the acquisition of the target posture information are synchronously aligned, so based on the internal reference information and the target posture information, the target object image in the image to be processed is fused with the virtual background image, and when the target image is obtained, the accuracy of the picture rendering is guaranteed, thereby improving the presentation effect of the video obtained by virtual production. Moreover, the present application does not need to obtain posture information with the help of a hardware locator, nor does it need to use professional-level camera equipment to solve the problem of synchronization between the hardware locator and the shot picture, thereby reducing the cost of virtual production and facilitating the widespread application of virtual production. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a movie screen provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a live broadcast screen provided in an embodiment of the present application;

[0051] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a green screen laying method provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of a green screen laying method provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of a green screen laying method provided in an embodiment of the present application;

[0055] Figure 8 A schematic diagram of a position reference pattern provided in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of a flow chart of a method for determining a camera posture provided in an embodiment of the present application;

[0057] Fig.10 A schematic diagram of a boundary line of a green screen provided in an embodiment of the present application;

[0058] Fig.11 A schematic diagram of a target imaging surface provided in an embodiment of the present application;

[0059] Fig.12 A flowchart of an image processing method provided in an embodiment of the present application;

[0060] Fig.13 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0061] Fig.14 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0064] Virtual production: refers to the video production method that combines real subjects with virtual backgrounds.

[0065] Green screen / blue screen cutout: refers to the cutout method using a green or blue screen.

[0066] Camera calibration: In the image measurement process and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are the camera parameters, and the process of solving the camera parameters is called camera calibration.

[0067] Camera pose estimation: refers to the method of obtaining the spatial position and attitude angle of the camera. Commonly used methods include QR code positioning, SLAM, radar wave positioning, multi-view camera pose detection, 3D reconstruction, etc.

[0068] QR code: A specific graphic that can be scanned to obtain its pattern encoding content.

[0069] solvePNP model: A model in OpenCV that finds the position (e.g., rotation, offset, etc.) of a camera in the world coordinate system based on multiple known point pairs. OpenCV is a cross-platform computer vision and machine learning software library.

[0070] The design concept of the embodiments of the present application is introduced below.

[0071] When performing virtual production, the relevant technology installs a locator on the shooting device to obtain the posture information of the shooting device, and uses the posture information to control the rendering of the virtual scene, and then superimposes the subject (people, objects, etc.) in the real scene captured by the shooting device onto the virtual scene.

[0072] However, the above method is prone to the situation where the time when the locator obtains the posture information is not synchronized with the time when the camera captures the picture. This will cause the virtual scene to not match the real subject, resulting in rendering errors in the picture, and then resulting in poor presentation effect of the video obtained by virtual production.

[0073] Through analysis, it is found that if the position information of the shooting device is determined directly based on the picture of the real scene shot by the shooting device, the synchronization problem between the locator and the shot picture can be effectively solved. In view of this, the embodiment of the present application provides the following image processing method, which includes:

[0074] An image to be processed taken by a real shooting device is obtained, wherein the image to be processed includes a target object image. If at least one position reference pattern is detected in the image to be processed, at least one position parameter pattern, each first position information in the image coordinate system, and each second position information in the world coordinate system are determined respectively. Then, based on each first position information, each second position information, and the internal reference information of the real shooting device, the target pose information of the real shooting device is determined. Then, based on the internal reference information and the target pose information, the target object image is fused with the virtual background image to obtain the target image.

[0075] In the embodiment of the present application, after obtaining the image to be processed shot by the real shooting device, the target posture information of the real shooting device is determined by the position reference pattern in the image to be processed, the first position information in the image coordinate system, and the second position information in the world coordinate system, so that the shooting of the image to be processed and the acquisition of the target posture information are synchronously aligned, so based on the internal reference information and the target posture information, the target object image in the image to be processed is fused with the virtual background image, and when the target image is obtained, the accuracy of the picture rendering is guaranteed, thereby improving the presentation effect of the video obtained by virtual production. Moreover, the present application does not need to obtain posture information with the help of a hardware locator, nor does it need to use professional-level camera equipment to solve the problem of synchronization between the hardware locator and the shot picture, thereby reducing the cost of virtual production and facilitating the widespread application of virtual production.

[0076] refer to Figure 1 , which is a system architecture diagram applicable to an embodiment of the present application. The system architecture includes at least a terminal device 101 and a server 102. The number of terminal devices 101 can be one or more, and the number of servers 102 can also be one or more. The present application does not specifically limit the number of terminal devices 101 and servers 102.

[0077] The terminal device 101 is pre-installed with a virtual production application, wherein the virtual production application is a client application, a web application, a small program application, etc. The terminal device 101 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart home appliance, a smart voice interaction device, a smart car device, etc., but is not limited thereto. The terminal device 101 can have a video shooting function, or can be connected to a real shooting device.

[0078] Server 102 is a background server for virtual production applications. Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 101 and the server 102 can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0079] The image processing method in the embodiment of the present application can be executed by the terminal device 101, can be executed by the server 102, or can be executed interactively by the terminal device 101 and the server 102.

[0080] For example, the server 102 executes the image processing method in the embodiment of the present application, which includes the following steps:

[0081] The terminal device 101 captures and obtains an image to be processed, wherein the image to be processed includes an image of a target object. The image to be processed is then sent to the server 102. The server 102 performs position reference pattern detection on the image to be processed. If at least one position reference pattern is detected in the image to be processed, at least one position parameter pattern is determined, each of which has its first position information in the image coordinate system, and each of which has its second position information in the world coordinate system. Then, based on each of the first position information, each of the second position information, and the internal reference information of the real shooting device, the target position information of the real shooting device is determined. Thereafter, based on the internal reference information and the target position information, the target object image is fused with the virtual background image to obtain the target image. The server 102 sends the target image to the terminal device 101, and the terminal device 101 displays the target image in the display interface.

[0082] In practical applications, the image processing method in the embodiments of the present application can be applied to virtual and real scenes such as film and television special effects, virtual live broadcast, game production, shopping, etc.

[0083] Application scenario 1: The image processing method in the embodiment of the present application can be applied to generate special effects in movies, such as Figure 2 As shown, the movie screen includes a screen 201 of real actors and a screen 202 of virtual scenes, which can reduce the cost and threshold of movie special effects production.

[0084] Application scenario 2: The image processing method in the embodiment of the present application can be applied to virtual live broadcast, such as Figure 3As shown, the live broadcast screen includes a screen 301 of the real anchor and a screen 302 of the virtual product. In the specific implementation, different virtual products can be switched according to actual needs, which brings convenience to the anchor. In addition, the live broadcast screen also includes virtual scenes, and different virtual scenes can be switched according to actual needs, without the need to redecorate the live broadcast room for each live broadcast theme, which brings convenience to the live broadcast and greatly reduces the cost of the live broadcast. In addition, feature effects can be added to the live broadcast screen according to actual needs, so as to bring better live broadcast screen effects and improve the audience's experience of watching live broadcasts.

[0085] based on Figure 1 The system architecture diagram shown in the figure, the embodiment of the present application provides a process of an image processing method, such as Figure 4 As shown, the process of the method is executed by a computer device, which may be Figure 1 The terminal device 101 and / or server 102 shown includes the following steps:

[0086] Step S401, obtaining an image to be processed shot by a real shooting device.

[0087] Specifically, the real shooting device can be a real camera, camcorder, camera or other shooting device, or it can be an electronic device with shooting function such as a mobile phone, tablet computer, smart watch or the like.

[0088] The image to be processed can be a single image (such as a photo) taken by a real shooting device; it can also be a video frame in a video to be processed taken by a real shooting device. Specifically, after the video to be processed is taken by the real shooting device, each video frame in the video to be processed can be used as the image to be processed to perform subsequent processing, or some video frames can be extracted from the video to be processed as the image to be processed to perform subsequent processing. This application does not make any specific limitations on this.

[0089] The image to be processed includes a target object image and a real background image, wherein the target object can be a person or an object.

[0090] Step S402: if at least one position reference pattern is detected in the image to be processed, at least one position parameter pattern is respectively determined, first position information of each pattern in the image coordinate system, and second position information of each pattern in the world coordinate system.

[0091] Specifically, the location reference pattern is a specific pattern embedded with coded content. The location reference pattern may be a QR code, a mini-program code, etc. The coded content may be the pattern number, pattern name, or other content that uniquely identifies the location reference pattern.

[0092] The image to be processed may be taken in a pre-built real scene including a background screen (green screen, blue screen, etc.), wherein the background screen includes at least one position reference pattern. Specifically, when building the real scene, the setting method of the background screen in the real scene and the arrangement method of the at least one position reference pattern in the background screen can be set according to actual needs.

[0093] For example, if Figure 5 As shown, the real scene includes two green screens, namely a blank green screen laid on the wall and a green screen containing 9 QR codes laid on the ground, wherein a real person stands on the green screen containing 9 QR codes.

[0094] For example, if Figure 6 As shown, the real scene includes three green screens, namely a blank green screen laid on the ground, and green screens each containing 4 QR codes laid on two walls, wherein real people stand on the blank green screen.

[0095] For example, if Figure 7 As shown, the real scene includes four green screens, namely a blank green screen laid on the ground, and green screens each containing 4 QR codes laid on three walls, wherein real people stand on the blank green screens.

[0096] It should be noted that the above-mentioned several configuration methods of the background curtain and the arrangement methods of the position reference patterns in the background curtain are only used for illustrative purposes, and the embodiments of the present application are not limited to the several exemplified methods.

[0097] Step S403: determining target position information of the real shooting device based on each piece of first position information, each piece of second position information, and internal reference information of the real shooting device.

[0098] Specifically, the intrinsic parameter information of the real shooting device is obtained by calibrating the real shooting device, wherein the calibration plate for obtaining feature points in the calibration can be a traditional checkerboard calibration plate or a background curtain with a position reference pattern. The intrinsic parameter information can refer to an intrinsic parameter matrix, through which the distorted image captured by the real shooting device can be corrected.

[0099] Taking a real camera as an example, the intrinsic parameter matrix of the camera is as follows:

[0100]

[0101] Among them, f x and f y represents the focal length in the x-axis and y-axis directions, c x and c y are the coordinates of the optical center on the x-axis and y-axis.

[0102] The target pose information of the real shooting device also refers to the external parameter information of the real shooting device, which can be represented by a pose matrix. After obtaining the target pose information of the real shooting device, a Kalman filter can be used to smooth the target pose information and remove jitter, thereby improving the accuracy of the target pose information.

[0103] Step S404: Based on the internal reference information and the target posture information, the target object image is fused with the virtual background image to obtain the target image.

[0104] Specifically, the virtual background image is a pre-generated virtual composite image. The real background image in the image to be processed is removed to obtain the target object image, and then the target object image is merged with the virtual background image to obtain the target image of the target object under the virtual background. It should be noted that the parameters such as the internal reference information and posture information involved in the embodiments of the present application can be stored on the blockchain and will not be described in detail later.

[0105] In the embodiment of the present application, after obtaining the image to be processed shot by the real shooting device, the target posture information of the real shooting device is determined by the position reference pattern in the image to be processed, the first position information in the image coordinate system, and the second position information in the world coordinate system, so that the shooting of the image to be processed and the acquisition of the target posture information are synchronously aligned, so based on the internal reference information and the target posture information, the target object image in the image to be processed is fused with the virtual background image, and when the target image is obtained, the accuracy of the picture rendering is guaranteed, thereby improving the presentation effect of the video obtained by virtual production. Moreover, the present application does not need to obtain posture information with the help of a hardware locator, nor does it need to use professional-level camera equipment to solve the problem of synchronization between the hardware locator and the shot picture, thereby reducing the cost of virtual production and facilitating the widespread application of virtual production.

[0106] Optionally, a pattern number is embedded in each position reference pattern, and each position reference pattern includes at least one feature point. At least one feature point on the position reference pattern is numbered in advance based on the pattern number of the position reference pattern and the position of at least one feature point on the position reference pattern in the position reference pattern to obtain a feature point number that uniquely identifies each feature point.

[0107] In the real world, the position of each position reference image on the background is pre-set, so the coordinates of the second feature point in the world coordinate system of the feature point on each position reference pattern can be known, where the second feature point coordinates are three-dimensional coordinates. In order to facilitate subsequent searches, the pattern number embedded in each position reference pattern, the feature point number corresponding to the feature point on each position reference pattern, and the second feature point coordinates are saved correspondingly, where the feature point on the position reference pattern can be a pre-set point, and the feature point numbers corresponding to the feature points on different position reference images are different.

[0108] For example, the position reference pattern is a two-dimensional code. Figure 8 As shown, the pattern number embedded in the QR code is 001, and the QR code includes 4 feature points, namely feature point A, feature point B, feature point C, and feature point D. Among them, the feature point number of feature point A is 0010, and the coordinates of the second feature point in the world coordinate system are (x 10 ,y 10 , z 10 ); the feature point number of feature point B is 0011, and the coordinates of the second feature point in the world coordinate system are (x 11 ,y 11 , z 11 ); the feature point number of feature point C is 0012, and the coordinates of the second feature point in the world coordinate system are (x 12 ,y 12 , z 12 ); the feature point number of feature point D is 0013, and the coordinates of the second feature point in the world coordinate system are (x 13 ,y 13 , z 13 ).

[0109] After obtaining the image to be processed shot by the real shooting device, the image to be processed is subjected to feature pattern detection to obtain feature point numbers corresponding to each of the multiple feature points and the first feature point coordinates of each in the image coordinate system, wherein the first feature point coordinates are two-dimensional coordinates. In addition, by decoding the position reference pattern, the pattern number embedded in the position reference pattern can be obtained.

[0110] If the number of the plurality of feature points is greater than or equal to the number of feature points contained in a position reference pattern, it is determined that at least one position reference pattern is detected in the image to be processed. At this time, at least one feature point contained in each position reference pattern detected, the corresponding feature point number, and the first feature point coordinates of each feature point in the image coordinate system are used as the first position information of the position reference pattern.

[0111] If the number of multiple feature points is less than the number of feature points contained in a position reference pattern, it is determined that no position reference pattern is detected in the image to be processed. Then, the historical posture information corresponding to the previous image to be processed taken by the real shooting device is used as the target posture information.

[0112] For example, the position reference pattern is set to be a two-dimensional code, and a two-dimensional code includes 4 feature points. The image to be processed is subjected to two-dimensional code detection to obtain two pattern numbers and the first feature point coordinates of the eight feature points in the image coordinate system, wherein the two pattern numbers are 001 and 002 respectively.

[0113] The eight feature points are feature point A, feature point B, feature point C, feature point D, feature point E, feature point F, feature point G, and feature point H. Feature point A is numbered 0010, feature point B is numbered 0011, feature point C is numbered 0012, feature point D is numbered 0013, feature point E is numbered 0020, feature point F is numbered 0021, feature point G is numbered 0022, and feature point H is numbered 0023.

[0114] Since the number of feature points detected is greater than 4, it is determined that at least one position reference pattern is detected in the image to be processed. From the feature point numbers, it can be seen that feature point A, feature point B, feature point C, and feature point D are feature points on a position reference pattern (pattern number is 001), and the feature point numbers corresponding to feature point A, feature point B, feature point C, and feature point D, as well as the first feature point coordinates of each in the image coordinate system, are used as the first position information of the position reference pattern.

[0115] If feature point E, feature point F, feature point G, and feature point H are feature points on another position reference pattern (pattern number is 002), then the feature point numbers corresponding to feature point E, feature point F, feature point G, and feature point H, as well as their respective first feature point coordinates in the image coordinate system, are used as the first position information of another position reference pattern.

[0116] It should be noted that the embodiments of the present application are not limited to determining whether a position reference pattern is detected by judging whether the number of feature points obtained by detection is greater than or equal to the number of feature points contained in a position reference pattern, but may also be other methods. For example, when the number of feature points obtained by detection is greater than or equal to the number of feature points contained in a position reference pattern, and the feature points obtained by detection contain at least one position reference pattern corresponding to all feature points, it is determined that at least one position reference pattern is detected in the image to be processed.

[0117] In the embodiment of the present application, feature point detection is performed on the image to be processed, and it is determined whether the position reference pattern is detected based on the number of feature points detected. When the position reference pattern is detected, the first position information of the position reference image in the image coordinate system is determined based on the first feature point coordinates of the feature points in the position reference pattern in the image coordinate system, and the first position information is used for the subsequent determination of the posture information of the real shooting device, thereby ensuring the real-time and accuracy of the posture information obtained. When the position reference pattern is not detected, the historical posture information corresponding to the previous image to be processed is used as the target posture information, thereby ensuring the continuity of the posture information of the real shooting device.

[0118] Optionally, for each position reference pattern of the at least one detected position reference pattern:

[0119] Based on the pattern number embedded in the position reference pattern obtained by detection, at least one feature point in the position reference pattern, the corresponding feature point number, and the second feature point coordinates of each feature point in the world coordinate system are obtained by querying. Then, the obtained feature point numbers and the corresponding second feature point coordinates are used as the second position information of the position reference pattern.

[0120] Specifically, the position reference pattern is decoded to obtain a pattern number embedded in the position reference pattern. Then, based on the pattern number of the position reference pattern, at least one feature point on the position reference pattern is queried to obtain the respective corresponding feature point numbers and respective second feature point coordinates in the world coordinate system.

[0121] For example, the position reference pattern is set to a QR code, and a QR code includes 4 feature points. Perform QR code detection on the image to be processed, and obtain two pattern numbers and the first feature point coordinates of the 8 feature points in the image coordinate system, where the two pattern numbers are: 001 and 002. Based on pattern number 001, feature point number 0010 (feature point A), feature point number 0011 (feature point B), feature point number 0012 (feature point C), feature point number 0013 (feature point D), and the second feature point coordinates of the feature points corresponding to each feature point number in the world coordinate system can be queried.

[0122] Based on pattern number 002, we can query and obtain feature point number 0020 (feature point E), feature point number 0021 (feature point F), feature point number 0022 (feature point G), feature point number 0023 (feature point H) and the second feature point coordinates of the feature points corresponding to each feature point number in the world coordinate system.

[0123] It should be noted that the embodiments of the present application are not limited to querying and obtaining at least one feature point in the position reference pattern based on the pattern number obtained by decoding, the corresponding feature point number, and the second feature point coordinates of each in the world coordinate system; it is also possible to query and obtain at least one feature point in the position reference pattern and the second feature point coordinates of each in the world coordinate system based on the feature point number obtained by detection after the position reference pattern is detected. The present application does not make any specific limitation on this.

[0124] In an embodiment of the present application, based on the pattern number obtained by decoding from the position reference pattern, the coordinates of the second feature point of the feature point on the position reference pattern in the world coordinate system are queried, and then the position of the position reference pattern in the world coordinate system is represented by the coordinates of the second feature point of the feature point on the position reference pattern in the world coordinate system, thereby improving the efficiency of obtaining the position of the position reference pattern in the world coordinate system.

[0125] Optionally, the embodiment of the present application uses at least the following method to determine the target pose information of the real shooting device:

[0126] For at least one position reference pattern, the following steps are performed respectively:

[0127] Based on the feature point number corresponding to at least one feature point contained in a position reference pattern, the first feature point coordinates and the second feature point coordinates corresponding to at least one feature point are grouped to obtain a feature point coordinate pair set corresponding to the position reference pattern, wherein each feature point coordinate pair in the feature point coordinate pair set includes a first feature point coordinate and a second feature point coordinate. Then, based on the feature point coordinate pair set corresponding to at least one position reference pattern and the internal parameter information of the real shooting device, the target posture information of the real shooting device is determined.

[0128] Specifically, the first feature point coordinates and the second feature point coordinates corresponding to the same feature point number are grouped into a group to obtain a feature point coordinate pair, and multiple feature point coordinate pairs constitute a feature point coordinate pair set. Then, all feature point coordinate pairs in the feature point coordinate pair set and the internal reference information of the real shooting device are input into the solvePNP model to obtain the target pose information of the real shooting device.

[0129] For example, see Fig. 9 , is a flow chart of a method for determining a camera posture provided in an embodiment of the present application, wherein the flow of the method is executed by a computer device, and the computer device may be Figure 1 The terminal device 101 and / or server 102 shown includes the following steps:

[0130] Step S901, obtaining an image to be processed taken by a camera.

[0131] Step S902 , performing two-dimensional code detection on the image to be processed, obtaining a pattern number, a feature point number corresponding to each of the at least one feature points, and the first feature point coordinates of each feature point in the image coordinate system.

[0132] For example, the detection obtains a pattern number, the feature numbers corresponding to the four feature points, and the first feature point coordinates of the four feature points in the image coordinate system, where the pattern number is: 001, and the four feature points are feature point A, feature point B, feature point C, and feature point D. The feature point number of feature point A is 0010, and the corresponding first feature point coordinates are (X 10 , Y 10 ); the feature point number of feature point B is 0011, and the corresponding first feature point coordinate is (X 11 , Y 11 ); the feature point number of feature point C is 0012, and the corresponding first feature point coordinate is (X 12 , Y 12 ); the feature point number of feature point D is 0013, and the corresponding first feature point coordinate is (X 13 , Y 13 ).

[0133] Step S903, determine whether the number of at least one feature point is greater than or equal to 4, if so, execute step S904, otherwise execute step S906.

[0134] Step S904: Based on the pattern number obtained by detection, query and obtain the feature point number corresponding to each of the at least one feature point and the second feature point coordinates of each feature point in the world coordinate system.

[0135] Specifically, based on the pattern number obtained by detection, the feature point number corresponding to at least one feature point and the second feature point coordinates of each feature point in the world coordinate system are queried from the pre-saved feature point numbers of all feature points and the second feature point coordinates of each feature point in the world coordinate system.

[0136] For example, based on pattern number 001, you can query to obtain feature point number 0010 (feature point A) and the corresponding second feature point coordinates (x 10 ,y 10 , z 10 ); feature point number 0011 (feature point B) and the corresponding second feature point coordinates (x 11 ,y 11 , z 11 ); feature point number 0012 (feature point C) and the corresponding second feature point coordinates (x 12 ,y 12 , z 12); feature point number 0013 (feature point D) and the corresponding second feature point coordinates (x 13 ,y 13 , z 13 ).

[0137] Step S905 , inputting the camera's intrinsic parameter matrix, the first feature point coordinates and the second feature point coordinates corresponding to at least one feature point into the solvePNP model, and outputting the target camera pose.

[0138] The first feature point coordinates and the second feature point coordinates are combined based on the feature point number to obtain four feature point coordinate pairs, which are feature point coordinate pairs (X 10 , Y 10 )-(x 10 ,y 10 , z 10 ), feature point coordinate pairs (X 11 , Y 11 )-(x 11 ,y 11 , z 11 ), feature point coordinate pairs (X 12 , Y 12 )-(x 12 ,y 12 , z 12 ), feature point coordinate pairs (X 13 , Y 13 )-(x 13 ,y 13 , z 13 ). The above four feature point coordinate pairs and the camera’s intrinsic parameter matrix are input into the solvePNP model, and the target camera pose is output.

[0139] Step S906, outputting the historical camera pose calculated for the last image to be processed.

[0140] In an embodiment of the present application, based on the feature points in the reference pattern, the coordinates of the first feature points in the image coordinate system and the coordinates of the second feature points in the world coordinate system, real-time pose detection is performed on the real shooting device to obtain the pose information of the real shooting device, thereby ensuring the synchronization between the real camera shooting picture and the obtained pose information, thereby improving the accuracy of the obtained pose information, and further improving the accuracy of subsequent virtual-real rendering.

[0141] Optionally, when an image to be processed is obtained by shooting in a real scene including a background curtain (green screen, blue screen, etc.), the real background image of the image to be processed sometimes includes not only the background curtain area, but may also include other interference areas outside the background curtain area (for example, objects outside the background curtain are photographed). At this time, if a traditional cutout algorithm is used to remove the background curtain area in the image to be processed, the cutout result will include the interference area image. At this time, if the cutout result is directly synthesized with the virtual background image, the obtained synthesized image will cause the synthesized image to have a glitch phenomenon, thereby affecting the presentation effect of the synthesized image.

[0142] In view of this, in an embodiment of the present application, the third position information of the boundary line between the green screen area and the non-green screen area in the real background image in the world coordinate system is obtained. Then, based on the third position information, the internal reference information and the target posture information, the fourth position information of the boundary line in the image coordinate system is determined. The image of the green screen area in the image to be processed is removed to obtain a preprocessed image, and then based on the fourth position information, the image of the non-green screen area outside the boundary line in the preprocessed image is removed to obtain the target object image.

[0143] Specifically, in the real world, the green screen is pre-set, so the third position information of the boundary line of the green screen in the world coordinate system can be known in advance, that is, the third position information of the boundary line between the green screen area and the non-green screen area in the real background image in the world coordinate system. In a specific implementation, the third position information of the boundary line in the world coordinate system can also refer to the position coordinates of each boundary point on the boundary line in the world coordinate system.

[0144] The third position information of the boundary line in the world coordinate system is transformed using the internal reference information and the target pose information to obtain the fourth position information of the boundary line in the image coordinate system. In a specific implementation, the fourth position information of the boundary line in the image coordinate system may also refer to the position coordinates of each boundary point on the boundary line in the image coordinate system.

[0145] A cutout algorithm is used to remove the image of the green screen area in the image to be processed to obtain a preprocessed image. Then, based on the fourth position information, the position of the boundary line in the preprocessed image is determined, and then the green screen area within the boundary line in the preprocessed image is set to non-transparent, and the non-green screen area outside the boundary line in the preprocessed image is set to fully transparent.

[0146] For example, if Fig.10 As shown, in the real world, the boundary line of the green screen includes 6 boundary points, namely boundary point a, boundary point b, boundary point c, boundary point d, boundary point e, and boundary point f. The intersection of the two walls and the ground is o, and the coordinates of boundary point a, boundary point b, boundary point c, boundary point d, boundary point e, boundary point f, and intersection point o in the world coordinate system are measured respectively.

[0147] Based on the camera's intrinsic parameter information and target pose information, the coordinates of the above-mentioned boundary points and intersection points in the world coordinate system are projected to the two-dimensional coordinates on the camera imaging surface, that is, the coordinates in the image coordinate system.

[0148] The image of the green screen area in the image to be processed is removed by using a cutout algorithm to obtain a preprocessed image. The preprocessed image is then initialized as a mask image with grayscale all equal to 0 (fully transparent). According to the coordinates of each boundary point and intersection point in the image coordinate system, the abof area (wall), bcdo area (wall), and odef area (ground) in the mask image are filled with grayscale 255 (opaque), and other areas in the mask image (non-green screen areas outside the boundary line) are kept fully transparent to obtain the target object image.

[0149] In an embodiment of the present application, based on the feature points in the reference pattern, the coordinates of the first feature points in the image coordinate system and the coordinates of the second feature points in the world coordinate system, real-time pose detection is performed on the real shooting device to obtain the real-time pose information of the real shooting device, and then based on the internal reference information and the real-time pose information, the boundary line of the green screen area in the image to be processed is predicted in real time, so that the non-green screen area in the image to be processed is completely cut out, ensuring excellent picture effects, and the effects are stable, robust and easy to use.

[0150] Optionally, the embodiment of the present application adopts at least the following method to fuse the target object image with the virtual background image to obtain the target image:

[0151] Based on the internal reference information, the field of view angle of the real shooting device is determined. Then, based on the internal reference information, the target posture information, and the field of view angle, the attribute information of the virtual shooting device in the virtual world is determined through the rendering engine. Based on the attribute information of the virtual shooting device and the reference position point in the target object image, the target position information of the target object image on the target imaging surface in the virtual world is determined. Based on the target position information of the target imaging surface, the target object image is fused with the virtual background image to obtain the target image.

[0152] Specifically, the following formula (1) and formula (2) are used to determine the field of view angle of the real shooting device based on the internal reference information:

[0153]

[0154] Among them, FOV x Represents the field of view angle in the x-axis direction in the real world, FOV x represents the field of view angle in the y-axis direction in the real world, W represents the width of the imaging surface in the image coordinate system in the real world, and H represents the height of the imaging surface in the image coordinate system in the real world.

[0155] The internal reference information, target posture information, and field of view angle are configured into the camera parameters in the 3D rendering engine to determine the attribute information of the virtual shooting device in the virtual world, wherein the attribute information of the virtual shooting device includes the internal reference information and posture information of the virtual camera in the virtual world.

[0156] The reference position point in the target object image can be set according to actual needs. For example, the sole of a person's foot, that is, the sole of the foot that contacts the ground. Another example is the point on a stool that contacts the ground.

[0157] Based on the attribute information of the virtual shooting device and the reference position point in the target object image, the target position information of the target imaging plane of the target object image in the virtual world is determined, wherein the target position information of the target imaging plane must satisfy: when the target object image is displayed in the target imaging plane, the positional relationship between the reference position point in the target object image and the virtual world satisfies a preset condition to ensure that the target object image is displayed at a reasonable position in the virtual world.

[0158] In a specific implementation, the target position information of the target imaging plane can be the distance between the target imaging plane and the virtual focus of the virtual shooting device. The optical center of the virtual shooting device passes through the optical axis of the virtual world, and the x and y axis directions of the virtual shooting device are the same as the x and y axis directions of the target imaging plane.

[0159] In an embodiment of the present application, based on the attribute information of the virtual shooting device and the reference position point in the target object image, the target position information of the target object image in the virtual world is determined, thereby improving the accuracy of locating the position of the target object image in the virtual world, thereby effectively improving the presentation effect of the image obtained by combining virtual and real.

[0160] Optionally, after obtaining the target position information of the target imaging surface, the target size information of the target imaging surface is determined based on the attribute information of the virtual shooting device and the target position information of the target imaging surface. Then, based on the target position information and the target size information, the target object image is fused with the virtual background image to obtain the target image.

[0161] Specifically, the virtual field of view of the virtual shooting device is obtained from the attribute information of the virtual shooting device, and then the target size information of the target imaging plane is determined based on the virtual field of view and the distance between the target imaging plane and the virtual focus of the virtual shooting device, as shown in the following formulas (3) and (4):

[0162]

[0163] Among them, fov x Indicates the field of view angle in the x-axis direction in the virtual world, fovy represents the field of view angle in the y-axis direction in the virtual world, w represents the width of the target imaging surface in the world coordinate system in the virtual world, and h represents the height of the target imaging surface in the world coordinate system in the virtual world.

[0164] Then, based on the target position information and the target size information, the target object image and the virtual background image are rendered in the rendering engine to obtain the target image.

[0165] For example, if Fig.11 As shown, after configuring the intrinsic parameter information, target pose information, and field of view angle of the real camera into the camera parameters in the 3D rendering engine, the pose information, field of view angle, and virtual focus k of the virtual camera in the virtual world can be determined. At the same time, the initial imaging plane W of the target person image Q in the virtual world and the display size of the target person image Q in the initial imaging plane W can be obtained. The position of the reference position point t in the initial imaging plane W is determined by the human skeleton point detection algorithm or by traversing the cutout results and selecting the position of the lowest opaque coordinate point.

[0166] Calculate the equation L of the line from the virtual focus k of the virtual camera to the reference position point t, and then determine the intersection point m of the line equation L and the ground of the virtual world. By adjusting the distance between the initial imaging plane W and the virtual focus k of the virtual camera, the size of the initial imaging plane W is adjusted accordingly. At the same time, the display size of the target person image Q in the initial imaging plane W is adjusted in the same proportion until the reference position point t is located at the intersection point m. Then calculate the distance D between the adjusted reference position point t (i.e., the intersection point m) and the virtual focus k.

[0167] The obtained initial imaging surface W is used as the target imaging surface N, and then the angle α between the straight line equation L and the optical axis of the virtual camera is calculated, and d = cos (α) * D is used as the distance between the target imaging surface N and the virtual focus k of the virtual camera. Then the obtained distance d is substituted into the above formulas (3) and (4) to obtain the width w and height h of the target imaging surface in the world coordinate system in the virtual world.

[0168] Based on the distance d, width w and height h, the target person image and the virtual background image are rendered in the rendering engine to obtain the target image.

[0169] In the embodiment of the present application, the target position information of the target imaging surface of the target object image in the virtual world is determined based on the attribute information of the virtual shooting device and the reference position point in the target object image. Then, based on the target position information of the target imaging surface, the target size information of the target imaging surface is correspondingly determined, so that the relationship between the target imaging surface and the virtual world is more matched, thereby improving the accuracy of the picture when the virtual and real are combined for rendering.

[0170] In order to better explain the embodiment of the present application, an image processing method provided by the embodiment of the present application is introduced below in combination with a specific implementation scenario. The process of the method can be as follows: Figure 1 The terminal device 101 shown in the figure may also be executed by the server 102, or the terminal device 101 and the server 102 may interact to execute the following steps: Fig.12 As shown:

[0171] Step S1201: camera calibration.

[0172] Specifically, a camera calibration is performed offline to obtain the intrinsic parameter information of the real camera.

[0173] Step S1202, obtaining a green screen video shot by a real camera.

[0174] Specifically, a real scene including a green screen is laid out in advance, and the green screen video is shot by a real camera in a real scene.

[0175] Step S1203: for each image to be processed in the green screen video, position reference pattern detection is performed on the image to be processed.

[0176] Specifically, each video frame in the green screen video is used as an image to be processed, and feature pattern detection is performed on the image to be processed to obtain feature point numbers corresponding to multiple feature points and the first feature point coordinates of each feature point in the image coordinate system, wherein the first feature point coordinates are two-dimensional coordinates.

[0177] Step S1204: camera posture detection.

[0178] If the number of the plurality of feature points is greater than or equal to the number of feature points included in one position reference pattern, it is determined that at least one position reference pattern is detected in the image to be processed.

[0179] For each position reference pattern of the at least one detected position reference pattern:

[0180] Based on the pattern number embedded in the position reference pattern obtained through detection, at least one feature point in the position reference pattern, the corresponding feature point number, and the second feature point coordinates of each feature point in the world coordinate system are obtained through query.

[0181] The first feature point coordinates and the second feature point coordinates corresponding to the same feature point number are divided into a group to obtain a feature point coordinate pair, and multiple feature point coordinate pairs constitute a feature point coordinate pair set. Then all feature point coordinate pairs in the feature point coordinate pair set and the camera's intrinsic parameter information are input into the solvePNP model to obtain the target pose information of the real camera.

[0182] If the number of the plurality of feature points is less than the number of feature points contained in a position reference pattern, it is determined that no position reference pattern is detected in the image to be processed. Then, the historical posture information corresponding to the previous image to be processed taken by the camera is used as the target posture information.

[0183] Step S1205, green screen cutout.

[0184] The third position information of the boundary line between the green screen area and the non-green screen area in the real background image in the world coordinate system is obtained. Then, based on the third position information, the intrinsic reference information and the target pose information, the fourth position information of the boundary line in the image coordinate system is determined. The image of the green screen area in the image to be processed is removed by using a cutout algorithm to obtain a preprocessed image, and then based on the fourth position information, the image of the non-green screen area outside the boundary line in the preprocessed image is removed to obtain the target object image.

[0185] Step S1206, obtaining a target image by rendering.

[0186] After configuring the intrinsic parameter information, target pose information, and field of view angle of the real camera into the camera parameters in the 3D rendering engine, the attribute information of the virtual shooting device in the virtual world can be determined. Based on the attribute information of the virtual shooting device and the reference position point in the target object image, the target position information of the target imaging surface of the target object image in the virtual world is determined. Then, based on the attribute information of the virtual shooting device and the target position information of the target imaging surface, the target size information of the target imaging surface is determined. Based on the target position information and the target size information, the target object image is fused with the virtual background image to obtain the target image.

[0187] In the embodiment of the present application, after obtaining the image to be processed shot by the real shooting device, the target posture information of the real shooting device is determined by the position reference pattern in the image to be processed, the first position information in the image coordinate system, and the second position information in the world coordinate system, so that the shooting of the image to be processed and the acquisition of the target posture information are synchronously aligned, so based on the internal reference information and the target posture information, the target object image in the image to be processed is fused with the virtual background image, and when the target image is obtained, the accuracy of the picture rendering is guaranteed, thereby improving the presentation effect of the video obtained by the virtual production. Moreover, the present application does not need to obtain posture information with the help of a hardware locator, nor does it need to use professional-level camera equipment to solve the problem of synchronization between the hardware locator and the shot picture, thereby reducing the cost of virtual production and facilitating the widespread application of virtual production. In addition, the algorithm complexity of the overall solution of the present application is very low, and it can be run in real time using a personal computer, and the pressure of deployment in the cloud is also small, which is very conducive to large-scale application.

[0188] Based on the same technical concept, the present application embodiment provides a structural schematic diagram of an image processing device, such as Fig.13 As shown, the device 1300 includes:

[0189] An acquisition module 1301 is used to acquire an image to be processed taken by a real shooting device, wherein the image to be processed includes an image of a target object;

[0190] A detection module 1302 is configured to determine, if at least one position reference pattern is detected in the image to be processed, the at least one position parameter pattern, first position information of each of the at least one position parameter pattern in the image coordinate system, and second position information of each of the at least one position parameter pattern in the world coordinate system;

[0191] A processing module 1303 is used to determine target position information of the real shooting device based on each first position information, each second position information, and internal reference information of the real shooting device;

[0192] The fusion module 1304 is used to fuse the target object image with the virtual background image based on the internal reference information and the target posture information to obtain a target image.

[0193] Optionally, the detection module 1302 is further configured to:

[0194] If no position reference pattern is detected in the image to be processed, the historical posture information corresponding to the previous image to be processed shot by the real shooting device is used as the target posture information.

[0195] Optionally, the detection module 1302 is specifically configured to:

[0196] For the at least one position reference pattern, the following steps are performed respectively:

[0197] At least one feature point contained in a position reference pattern obtained by detection, the corresponding feature point numbers, and the first feature point coordinates of each feature point in the image coordinate system are used as the first position information of the position reference pattern.

[0198] Optionally, the detection module 1302 is specifically configured to:

[0199] Based on the pattern number embedded in the one position reference pattern obtained by detection, query and obtain the at least one feature point, the corresponding feature point number, and the second feature point coordinates of each feature point in the world coordinate system;

[0200] The obtained characteristic point numbers and the corresponding second characteristic point coordinates are used as the second position information of the one position reference pattern.

[0201] Optionally, the processing module 1303 is specifically configured to:

[0202] For the at least one position reference pattern, the following steps are performed respectively:

[0203] Based on the feature point number corresponding to each of the at least one feature point included in a position reference pattern, first feature point coordinates and second feature point coordinates corresponding to each of the at least one feature point are grouped to obtain a feature point coordinate pair set corresponding to the one position reference pattern, wherein each feature point coordinate pair in the feature point coordinate pair set includes a first feature point coordinate and a second feature point coordinate;

[0204] Based on the feature point coordinate pair sets corresponding to each of the at least one position reference patterns and the internal parameter information of the real shooting device, the target position and pose information of the real shooting device is determined.

[0205] Optionally, the image to be processed also includes a real background image;

[0206] Also includes a screenshot module 1305;

[0207] The screenshot module 1305 is specifically used for:

[0208] Based on the internal reference information and the target pose information, the target object image is fused with the virtual background image, and before obtaining the target image, the boundary line between the green screen area and the non-green screen area in the real background image is obtained, and the third position information in the world coordinate system is obtained;

[0209] Determine fourth position information of the boundary line in the image coordinate system based on the third position information, the internal reference information and the target pose information;

[0210] Removing the image of the green screen area from the image to be processed to obtain a preprocessed image;

[0211] Based on the fourth position information, the image of the non-green screen area outside the boundary line in the preprocessed image is removed to obtain the target object image.

[0212] Optionally, the fusion module 1304 is specifically used for:

[0213] Based on the internal reference information, determining the field of view angle of the real shooting device;

[0214] Based on the internal reference information, the target posture information, and the field of view angle, determining attribute information of a virtual shooting device in a virtual world through a rendering engine;

[0215] Determining target position information of a target imaging plane of the target object image in the virtual world based on the attribute information of the virtual shooting device and the reference position point in the target object image;

[0216] Based on the target position information of the target imaging surface, the target object image is fused with the virtual background image to obtain a target image.

[0217] Optionally, the fusion module 1304 is specifically used for:

[0218] Determining target size information of the target imaging plane based on the attribute information of the virtual shooting device and the target position information of the target imaging plane;

[0219] Based on the target position information and the target size information, the target object image is fused with a virtual background image to obtain a target image.

[0220] In the embodiment of the present application, after obtaining the image to be processed shot by the real shooting device, the target posture information of the real shooting device is determined by the position reference pattern in the image to be processed, the first position information in the image coordinate system, and the second position information in the world coordinate system, so that the shooting of the image to be processed and the acquisition of the target posture information are synchronously aligned, so based on the internal reference information and the target posture information, the target object image in the image to be processed is fused with the virtual background image, and when the target image is obtained, the accuracy of the picture rendering is guaranteed, thereby improving the presentation effect of the video obtained by the virtual production. Moreover, the present application does not need to obtain posture information with the help of a hardware locator, nor does it need to use professional-level camera equipment to solve the problem of synchronization between the hardware locator and the shot picture, thereby reducing the cost of virtual production and facilitating the widespread application of virtual production. In addition, the algorithm complexity of the overall solution of the present application is very low, and it can be run in real time using a personal computer, and the pressure of deployment in the cloud is also small, which is very conducive to large-scale application.

[0221] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 1 The terminal device and / or server shown, such as Fig.14 As shown, it includes at least one processor 1401 and a memory 1402 connected to the at least one processor. The specific connection medium between the processor 1401 and the memory 1402 is not limited in the embodiment of the present application. Fig.14 For example, the processor 1401 and the memory 1402 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0222] In the embodiment of the present application, the memory 1402 stores instructions that can be executed by at least one processor 1401, and the at least one processor 1401 can perform the steps of the above-mentioned image processing method by executing the instructions stored in the memory 1402.

[0223] Among them, the processor 1401 is the control center of the computer device, and can use various interfaces and lines to connect various parts of the computer device, and realize virtual production by running or executing instructions stored in the memory 1402 and calling data stored in the memory 1402. Optionally, the processor 1401 may include one or more processing units, and the processor 1401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1401. In some embodiments, the processor 1401 and the memory 1402 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0224] Processor 1401 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0225] Memory 1402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1402 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. Memory 1402 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 1402 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0226] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned image processing method.

[0227] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned image processing method.

[0228] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0229] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0230] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0231] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0232] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0233] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be processed that is captured by a real shooting device in a pre-built real scene including a background curtain, wherein the background curtain includes at least one position reference pattern, the position reference pattern is a specific pattern embedded with coded content, and the image to be processed includes a target object image; If at least one position reference pattern is detected in the image to be processed, respectively determining the at least one position parameter pattern, first position information of each pattern in the image coordinate system, and second position information of each pattern in the world coordinate system; Performing a camera calibration offline to obtain the intrinsic parameter information of the real shooting device; Based on each first position information, each second position information, and internal reference information of the real shooting device, target pose information of the real shooting device is determined, and shooting the image to be processed and obtaining the target pose information are synchronously aligned; Based on the internal reference information and the target pose information, the target object image is fused with a virtual background image to obtain a target image.

2. The method according to claim 1, characterized in that Also includes: If no position reference pattern is detected in the image to be processed, the historical posture information corresponding to the previous image to be processed shot by the real shooting device is used as the target posture information.

3. The method according to claim 1, characterized in that The determining of the first position information of each of the at least one position parameter pattern in the image coordinate system comprises: For the at least one position reference pattern, the following steps are performed respectively: At least one feature point contained in a position reference pattern obtained by detection, the corresponding feature point numbers, and the first feature point coordinates of each feature point in the image coordinate system are used as the first position information of the position reference pattern.

4. The method according to claim 3, characterized in that The determining of the second position information of each of the at least one position parameter pattern in the world coordinate system comprises: Based on the pattern number embedded in the one position reference pattern obtained by detection, query and obtain the at least one feature point, the corresponding feature point number, and the second feature point coordinates of each feature point in the world coordinate system; The obtained characteristic point numbers and the corresponding second characteristic point coordinates are used as the second position information of the one position reference pattern.

5. The method according to claim 4, characterized in that The determining the target position information of the real shooting device based on each first position information, each second position information, and the internal reference information of the real shooting device includes: For the at least one position reference pattern, the following steps are performed respectively: Based on the feature point number corresponding to each of the at least one feature point included in a position reference pattern, first feature point coordinates and second feature point coordinates corresponding to each of the at least one feature point are grouped to obtain a feature point coordinate pair set corresponding to the one position reference pattern, wherein each feature point coordinate pair in the feature point coordinate pair set includes a first feature point coordinate and a second feature point coordinate; Based on the feature point coordinate pair sets corresponding to each of the at least one position reference patterns and the internal parameter information of the real shooting device, the target position and pose information of the real shooting device is determined.

6. The method according to claim 1, characterized in that The image to be processed also includes a real background image; Before fusing the target object image with the virtual background image based on the internal reference information and the target pose information to obtain the target image, the method further includes: Acquire third position information of a boundary line between a green screen area and a non-green screen area in the real background image in the world coordinate system; Determine fourth position information of the boundary line in the image coordinate system based on the third position information, the internal reference information and the target pose information; Removing the image of the green screen area from the image to be processed to obtain a preprocessed image; Based on the fourth position information, the image of the non-green screen area outside the boundary line in the preprocessed image is removed to obtain the target object image.

7. The method according to claim 1, characterized in that The step of fusing the target object image with the virtual background image based on the internal reference information and the target pose information to obtain the target image includes: Based on the internal reference information, determining the field of view angle of the real shooting device; Based on the internal reference information, the target posture information, and the field of view angle, determining attribute information of a virtual shooting device in a virtual world through a rendering engine; Determining target position information of a target imaging plane of the target object image in the virtual world based on the attribute information of the virtual shooting device and the reference position point in the target object image; Based on the target position information of the target imaging surface, the target object image is fused with the virtual background image to obtain a target image.

8. The method according to claim 7, characterized in that The step of fusing the target object image with the virtual background image based on the target position information of the target imaging surface to obtain the target image includes: Determining target size information of the target imaging plane based on the attribute information of the virtual shooting device and the target position information of the target imaging plane; Based on the target position information and the target size information, the target object image is fused with a virtual background image to obtain a target image.

9. An image processing device, characterized in that: include: An acquisition module is used to acquire an image to be processed that is captured by a real shooting device in a pre-built real scene including a background curtain, wherein the background curtain includes at least one position reference pattern, and the position reference pattern is a specific pattern embedded in the coded content, and the image to be processed includes a target object image; A detection module, configured to respectively determine the at least one position parameter pattern, first position information of each of the at least one position parameter pattern in the image coordinate system, and second position information of each of the at least one position parameter pattern in the world coordinate system if at least one position reference pattern is detected in the image to be processed; Performing a camera calibration offline to obtain the intrinsic parameter information of the real shooting device; A processing module, for determining target pose information of the real shooting device based on each first position information, each second position information, and internal reference information of the real shooting device, wherein shooting the image to be processed and obtaining the target pose information are synchronously aligned; A fusion module is used to fuse the target object image with the virtual background image based on the internal reference information and the target posture information to obtain a target image.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 8.

Images