Posture processing method, device, electronic device, storage medium and program product

Abstract

The present application provides a method, device, electronic device, computer-readable storage medium and computer program product for processing the posture of a camera; the present application relates to augmented reality and electronic map technology, and the method provided by the present application includes: obtaining an initial image including a first tracking frame, wherein the initial image is acquired by a camera when the first tracking frame is first positioned; based on the initial content of the first tracking frame of the initial image, training a target detector for identifying the initial content; obtaining a current image, wherein the current image is acquired in real time by a camera; performing positioning processing on the current image based on the initial content by the target detector to obtain a second tracking frame corresponding to the first tracking frame in the current image; and determining the current target posture of the camera in the world coordinate system based on the second tracking frame corresponding to the first tracking frame in the current image. Through the present application, the accuracy of determining the current target posture of the camera can be improved by repositioning.

Description

Technical Field

[0001] The present application relates to augmented reality and electronic map technology, and in particular to a camera posture processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] With the development of Internet technology and the diversification of human-computer interaction technology, the augmented reality function of video is widely used in various fields such as video, games, and user interface interaction. For example, the placement of virtual objects in augmented reality is adopted by multiple applications as a new interaction method.

[0003] In the related art, due to the instability of the user's control over the camera, or because the user himself will perform motion capture, the current position of the camera will change, resulting in instability in the implementation of the augmented reality interaction function. For example, the placed virtual objects have low stability in the real scene, and the virtual objects cannot be kept at the initial placement point, and are prone to drift and jitter with the movement of the camera. The accuracy of determining and calculating the current target position of the camera in the related art is low, which results in the inability to guarantee the stability of the implementation of the augmented reality interaction function when rendering based on the current target position. Summary of the invention

[0004] The embodiments of the present application provide a camera posture processing method, device, electronic device, computer-readable storage medium and computer program product, which can improve the accuracy of determining the current target posture of the camera by repositioning.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method for processing the position and posture of a camera, including:

[0007] Acquire an initial image including a first tracking frame, wherein the initial image is acquired when the camera is first positioned to the first tracking frame;

[0008] Based on the initial content of the first tracking box of the initial image, training an object detector for recognizing the initial content;

[0009] Acquire a current image, wherein the current image is acquired in real time by the camera within the acquisition period;

[0010] performing positioning processing on the current image based on the initial content by the object detector to obtain a second tracking frame in the current image corresponding to the first tracking frame;

[0011] Based on a second tracking frame in the current image corresponding to the first tracking frame, a current target pose of the camera in a world coordinate system is determined.

[0012] The present application provides a camera posture processing device, including:

[0013] An initial module, configured to obtain an initial image including a first tracking frame, wherein the initial image is acquired when the camera is first positioned to the first tracking frame;

[0014] A training module, configured to train a target detector for identifying the initial content based on the initial content of the first tracking frame of the initial image;

[0015] A current module is used to obtain a current image, wherein the current image is acquired in real time by the camera;

[0016] a re-positioning module, configured to perform positioning processing on the current image based on the initial content through the object detector, so as to obtain a second tracking frame in the current image corresponding to the first tracking frame;

[0017] A determination module is used to determine a current target pose of the camera in a world coordinate system based on a second tracking frame corresponding to the first tracking frame in the current image.

[0018] In the above scheme, before acquiring the initial image including the first tracking frame, the initial module is also used to: in response to a click operation on the target position, perform any one of the following processing: determine a tracking frame with the target position as the center point and a set size as the first tracking frame; determine a tracking frame with the target position as a set corner point and a set size as the first tracking frame.

[0019] In the above scheme, the training module is also used to: when the test result of the target detector is less than the test result threshold and the cumulative number of batch training of the target detector does not exceed the number threshold, perform the following processing: batch training the target detector based on a training sample set, wherein the training sample set includes training samples corresponding to the initial content; and add the recognition results obtained based on the training sample set during the batch training to the training sample set to update the training sample set.

[0020] In the above scheme, the determination module is also used to: determine the current posture of the first tracking frame in the world coordinate system; determine the current posture of the first tracking frame in the camera coordinate system based on the second tracking frame corresponding to the first tracking frame and the first tracking frame; determine the current target posture of the camera in the world coordinate system based on the current posture of the first tracking frame in the world coordinate system and the current posture of the first tracking frame in the camera coordinate system.

[0021] In the above scheme, the determination module is also used to: perform tracking processing on the current image based on the third tracking frame corresponding to the first tracking frame in the previous frame image of the current image to obtain a second tracking frame corresponding to the third tracking frame in the current image; when the position difference between the second tracking frame corresponding to the third tracking frame and the second tracking frame corresponding to the first tracking frame is less than a position difference threshold, perform smoothing correction processing on the current position of the camera in the world coordinate system so that the camera moves from the current position to the current target position corresponding to the current target posture in the world coordinate system; when the position difference between the second tracking frame corresponding to the third tracking frame and the second tracking frame corresponding to the first tracking frame is not less than the position difference threshold, update the camera from the current position to the current target position corresponding to the current target posture in the world coordinate system.

[0022] In the above scheme, the determination module is also used to: determine the current depth of the first tracking frame in the camera coordinate system based on the side length of the first tracking frame, the current physical focal length of the camera and the size of the second tracking frame corresponding to the first tracking frame; determine the current posture of the first tracking frame in the camera coordinate system based on the current depth of the first tracking frame in the camera coordinate system, the internal parameters of the camera and the current position of the second tracking frame corresponding to the first tracking frame in the world coordinate system; and combine the current position of the first tracking frame in the camera coordinate system and the current posture of the first tracking frame in the camera coordinate system to form the current posture of the first tracking frame in the camera coordinate system.

[0023] In the above scheme, the determination module is also used to: determine the initial pose of the first tracking frame in the camera coordinate system; determine the initial pose of the camera in the world coordinate system; and determine the current pose of the first tracking frame in the world coordinate system based on the initial pose of the camera in the world coordinate system and the initial pose of the first tracking frame in the camera coordinate system.

[0024] In the above scheme, the determination module is also used to: determine the initial position of the camera in the world coordinate system; obtain the initial gyroscope data of the camera corresponding to the first tracking frame, and determine the initial posture of the camera corresponding to the initial gyroscope data; and combine the initial position of the camera in the world coordinate system and the initial posture of the camera to form the initial posture of the camera in the world coordinate system.

[0025] In the above scheme, the determination module is also used to: determine the initial depth of the first tracking frame in the camera coordinate system based on the horizontal plane normal vector, the internal parameters of the camera and the initial position of the first tracking frame in the world coordinate system; determine the initial posture of the first tracking frame in the camera coordinate system based on the initial depth of the first tracking frame in the camera coordinate system, the internal parameters of the camera and the initial position of the first tracking frame in the world coordinate system.

[0026] In the above scheme, the repositioning module is also used to: obtain current gyroscope data when the current image is captured by the camera; when the movement speed of the camera is not greater than a first speed threshold and the training process of the target detector is completed, the current image is positioned based on the initial content by the target detector to obtain a second tracking frame in the current image corresponding to the first tracking frame.

[0027] In the above scheme, the determination module is also used to: when the movement speed of the camera is greater than the first speed threshold, determine the current posture of the camera in the world coordinate system based on the current gyroscope data of the camera; and combine the current posture of the camera in the world coordinate system and the current position of the camera in the world coordinate system to form the current target posture of the camera in the world coordinate system.

[0028] In the above scheme, the determination module is also used for: when any one of the following conditions is met, based on the third tracking frame corresponding to the first tracking frame in the previous frame image of the current image, tracking processing is performed on the current image to obtain a second tracking frame corresponding to the third tracking frame in the current image: the movement speed of the camera is not greater than the first speed threshold, and the training process of the target detector has not yet ended; the movement speed of the camera is not greater than the first speed threshold, the training process of the target detector is ended, and the overlapping area between the second tracking frame corresponding to the first tracking frame and the first tracking frame is less than the overlapping area threshold; obtaining a tracking score corresponding to the tracking processing, and obtaining a tracking score threshold that is negatively correlated with the movement speed of the camera; when the tracking score exceeds the tracking score threshold, determining the current target posture of the camera in the world coordinate system based on the second tracking frame corresponding to the third tracking frame in the current image.

[0029] In the above scheme, the determination module is also used to: when the tracking score does not exceed the tracking score threshold, determine the current posture of the camera in the world coordinate system based on the current gyroscope data of the camera; and combine the current posture of the camera in the world coordinate system and the current position of the camera in the world coordinate system to form the current target posture of the camera in the world coordinate system.

[0030] In the above scheme, the determination module is also used to: determine the current posture of the third tracking frame in the world coordinate system; determine the current posture of the third tracking frame in the camera coordinate system based on the second tracking frame corresponding to the third tracking frame in the current image, and the third tracking frame; determine the current target posture of the camera in the world coordinate system based on the current posture of the third tracking frame in the world coordinate system and the current posture of the third tracking frame in the camera coordinate system.

[0031] In the above scheme, the determination module is also used to: determine the current depth of the third tracking frame in the camera coordinate system based on the side length of the third tracking frame, the current physical focal length of the camera and the size of the second tracking frame corresponding to the third tracking frame; determine the current posture of the third tracking frame in the camera coordinate system based on the current depth of the third tracking frame in the camera coordinate system, the internal parameters of the camera and the current position of the second tracking frame corresponding to the third tracking frame in the world coordinate system; and combine the current position of the third tracking frame in the camera coordinate system and the current posture of the camera in the camera coordinate system to form the current posture of the third tracking frame in the camera coordinate system.

[0032] In the above scheme, before determining the current target pose of the camera in the world coordinate system based on the second tracking frame corresponding to the first tracking frame in the current image, the determination module is also used to: when the second tracking frame corresponding to the first tracking frame is located in the boundary area of ​​the current image, obtain a fourth tracking frame of the current image; wherein the fourth tracking frame is adjacent to the second tracking frame and is closer to the center position of the current image than the second tracking frame; and determine the fourth tracking frame as the second tracking frame.

[0033] An embodiment of the present application provides an electronic device, including:

[0034] A memory for storing executable instructions;

[0035] The processor is used to implement the camera posture processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0036] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing the camera posture processing method provided in the embodiment of the present application when executed by a processor.

[0037] An embodiment of the present application provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, the camera posture processing method provided in the embodiment of the present application is implemented.

[0038] The embodiments of the present application have the following beneficial effects:

[0039] By learning the initial content of the first tracking frame, a target detector for identifying the initial content is trained, so that the initial content can be identified in the current image collected in real time, so as to locate the second tracking frame corresponding to the first tracking frame, and then determine the current target pose of the camera in the world coordinate system according to the second tracking frame. Since the second tracking frame is obtained based on the initial content positioning, the current target pose determined based on the second tracking frame matches the camera pose when the initial image is collected, thereby improving the accuracy of camera pose repositioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a structural schematic diagram of a camera posture processing system provided in an embodiment of the present application;

[0041] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0042] Figures 3A-3C It is a flowchart of a method for processing the position and posture of a camera provided in an embodiment of the present application;

[0043] Figures 4A-4B It is a display interface diagram of the camera posture processing method provided in an embodiment of the present application;

[0044] Figure 5 is an algorithm structure diagram of a camera posture processing method provided in an embodiment of the present application;

[0045] Figures 6A-6G It is a logical schematic diagram of the camera posture processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0047] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0048] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0051] 1) Camera coordinate system: The camera coordinate system is a three-dimensional rectangular coordinate system with the focal center of the camera as the origin and the optical axis as the z-axis. The x-axis of the camera coordinate system is parallel to the x-axis of the image coordinate system of the captured image, and the y-axis of the camera coordinate system is parallel to the y-axis of the image coordinate system of the captured image.

[0052] 2) Image coordinate system: The image coordinate system is a coordinate system with the upper left vertex of the image captured by the camera as the coordinate origin. The x-axis and y-axis of the image coordinate system are the length and width directions of the captured image.

[0053] 3) World Coordinate System: The world coordinate system can describe the position of the camera in the real world, and can also describe the position of objects in the image captured by the camera in the real world. Usually, the x-axis of the world coordinate system points horizontally to the east, the y-axis points horizontally to the north, and the z-axis points vertically upward.

[0054] 4) Posture: Posture is used to describe position and posture. Position can be expressed by coordinates in various coordinate systems, and posture refers to the orientation in various coordinate systems, which can be expressed by rotation matrices.

[0055] 5) Inertial measurement unit (IMU): IMU is a device used to measure the three-axis attitude angle (or angular rate) and acceleration of an object. IMU usually contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometer is used to detect the acceleration signal of the object in the independent three axes of the carrier coordinate system, and the gyroscope is used to detect the angular velocity signal of the carrier relative to the navigation coordinate system. IMU can measure the angular velocity and acceleration of an object in three-dimensional space, and use this to determine the attitude of the object.

[0056] 6) Computer Vision Technology (CV): Computer vision refers to the use of cameras and computers to replace human eyes to identify and measure objects, and further process images to make the computer processing more suitable for human observation or transmission to instruments for detection. Computer vision technology usually includes image processing, image recognition, image semantic understanding, video processing, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, etc.

[0057] 7) Augmented Reality (AR) is one of the hottest technologies in recent years. The client "enhances" the displayed real world by overlaying and displaying AR models on the video frames collected by the camera in real time. When the client displays the AR model, it needs to first determine the position of the camera in the world coordinate system.

[0058] In order to reduce jitter and improve stability when implementing augmented reality functions, the related technology will adopt a positioning method that combines an image block tracking algorithm with a camera gyroscope. Specifically, in response to a user's click operation on a target position in the human-computer interaction interface, an image area block including the target position is recorded, and a visual tracking algorithm, such as optical flow tracking, is used to track the image area block, and the position and posture of the camera are calculated in real time, thereby ensuring the stability of the augmented reality function. For example, the augmented reality function is to place a virtual model at the target position of the captured image, thereby ensuring the stability of the virtual model.

[0059] In the related technology, a visual image block tracking algorithm is usually used to position the camera. Since the user will control the camera for a long time or use a fast motion acquisition mode, the blurry visual image during the acquisition cycle will cause image tracking failure, and the current position of the camera cannot be accurately determined. Therefore, when rendering virtual objects according to the current position of the camera, the virtual objects will gradually move away from the target position obtained by the user's click, making the placement of the virtual objects in the acquisition screen unrealistic, moving and drifting with the movement of the camera, and unable to be fixed at the target position, affecting the user's augmented reality experience.

[0060] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for processing the posture of a camera, which can identify the second tracking frame corresponding to the initial first tracking frame in the current image through an object detector, and recalculate the current posture of the camera based on the second tracking frame, so as to accurately determine the current posture of the camera. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a laptop computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a vehicle-mounted terminal, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, an intelligent voice interaction device, a smart home appliance, an aircraft). The following describes an exemplary application when the device is implemented as a terminal.

[0061] See also Figure 1 , Figure 1 It is a structural diagram of the camera posture processing system provided in an embodiment of the present application. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0062] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), referred to as IVICS, is a development direction of Intelligent Transportation Systems (ITS). IVICS uses advanced wireless communications and new generation Internet technologies to implement all-round dynamic real-time information interaction between vehicles and roads, and conducts active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thus forming a safe, efficient and environmentally friendly road traffic system.

[0063] In some embodiments, the camera posture processing method provided in the embodiment of the present application can be implemented collaboratively by the terminal and the server. The camera posture processing method provided in the embodiment of the present application can be applied to an electronic map APP. The electronic map APP will provide an AR-based map filling function. For example, in response to a user's click operation, a virtual model, such as a building, is filled in the collected original map. The processor of the terminal 400 receives an image stream collected by the camera (the camera can be set in the terminal or independent of the terminal). In response to the user's click operation on the target position in the human-computer interaction interface of the terminal 400, the terminal renders a virtual model based on the target position, determines a first tracking frame based on the target position and an initial image collected when the first tracking frame is determined, and the terminal 400 sends a first tracking frame to the server 200. The server 200 trains a target detector for identifying the initial content based on the initial content of the first tracking frame of the initial image. During the training, the processor of the terminal 400 is still receiving the image stream captured by the camera. The terminal 400 obtains the current image currently captured by the camera in real time, and also sends the current image to the server 200. The server 200 performs positioning processing on the current image based on the initial content through the target detector, obtains the second tracking frame corresponding to the first tracking frame in the current image, and returns the second tracking frame in the current image to the terminal 400. The terminal 400 determines the current target pose of the camera in the world coordinate system, and then the terminal 400 corrects the position of the camera based on the current target pose of the camera in the world coordinate system, and renders the virtual model based on the corrected camera position.

[0064] In some embodiments, the camera posture processing method provided in the embodiments of the present application can be implemented independently by a terminal or a server. When implemented independently by a terminal, the processor of the terminal receives an image stream captured by the camera. In response to a user's click operation on a target position in a human-computer interaction interface of the terminal, the terminal renders a virtual model based on the target position, determines a first tracking frame based on the target position and an initial image captured when determining the first tracking frame, and trains a target detector for identifying the initial content based on the initial content of the first tracking frame of the initial image. During the training, the processor of the terminal is still receiving the image stream captured by the camera. The terminal obtains the current image currently captured by the camera in real time. The terminal performs positioning processing on the current image based on the initial content through the target detector to obtain a second tracking frame corresponding to the first tracking frame in the current image. The terminal determines the current target posture of the camera in the world coordinate system, and then the terminal corrects the position of the camera based on the current target posture of the camera in the world coordinate system, and renders the virtual model based on the corrected camera position.

[0065] In some embodiments, the terminal or server can implement the camera posture processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP, an instant messaging APP, or an electronic map APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0066] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present invention.

[0067] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0068] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0069] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0070] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0071] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0072] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0073] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0074] A network communication module 452, for reaching other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0075] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0076] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0077] In some embodiments, the camera posture processing device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 The camera posture processing device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, including the following software modules: an initial module 4551, a training module 4552, a current module 4553, a repositioning module 4554 and a determination module 4555. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0078] The camera posture processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0079] See also Figure 3A , Figure 3A is a flow chart of the camera posture processing method provided in the embodiment of the present application, which will be combined with Figure 3A Steps 101 to 105 are shown for explanation.

[0080] In step 101, an initial image including a first tracking frame is acquired.

[0081] As an example, the initial image is acquired by the camera when it is positioned at the first tracking frame. The first tracking frame may be manually determined based on a click operation of the user, or the first tracking frame may be automatically determined based on an autofocus function of the camera.

[0082] In some embodiments, see Figures 4A-4B , Figures 4A-4B It is a display interface diagram of the camera posture processing method provided by the embodiment of the present application. Before acquiring the initial image including the first tracking frame, in response to the click operation on the target position, any one of the following processes is performed: a tracking frame with the target position as the center point and a set size is determined as the first tracking frame; a tracking frame with the target position as the set corner point and a set size is determined as the first tracking frame. After receiving the click operation on the target position, the selected virtual objects 401A and 401B can be placed at the target position. The method of determining the first tracking frame by the click operation can realize the user's manual focus or manual object placement, so that the focus position or object placement position meets the user's needs and improves the user experience in the process of using the AR function.

[0083] In some embodiments, the camera posture processing method provided in the embodiments of the present application can be applied to AR observation scenes. After the camera automatically focuses, in order to ensure the stability of the observation scene, the autofocus result can be used as the first tracking frame, so that in the subsequent observation process, real-time repositioning based on the first tracking frame can be performed to ensure the stability of the observation scene.

[0084] In step 102, based on the initial content of the first tracking frame of the initial image, an object detector for recognizing the initial content is trained.

[0085] In some embodiments, in step 102, based on the initial content of the first tracking frame of the initial image, training the target detector for identifying the initial content can be implemented by the following technical solution: when the test result of the target detector is less than the test result threshold, and the cumulative number of batch training of the target detector does not exceed the number threshold, the following processing is performed: batch training is performed on the target detector based on the training sample set, wherein the training sample set includes training samples corresponding to the initial content; the recognition results obtained based on the training sample set during the batch training are added to the training sample set to update the training sample set. Updating the obtained recognition results to the training sample set during the training process can not only expand the training sample set, but also improve the effectiveness of the training samples, thereby improving the training effect.

[0086] As an example, the training process of the target detector can be carried out in a variety of ways, such as a kernel correlation filtering method. The training goal is to train a target detector for tracking frame detection, and the target detector can detect a tracking frame including the content of the first tracking frame. When training the target detector, the area of ​​the first tracking frame is usually selected as a positive sample, and the surrounding area of ​​the first tracking frame is selected as a negative sample. The surrounding area can be an adjacent area of ​​the first tracking frame, and the adjacent area can be completely independent of the first tracking frame, or the overlapping ratio of the adjacent area and the first tracking frame is less than a ratio threshold. During forward propagation, the target detector is used to detect the next frame image to obtain a second tracking frame corresponding to the first tracking frame in the next frame image, and then the area of ​​the second tracking frame is used to update the training sample set to continue training the target detector, wherein, based on the matching degree with the first tracking frame, the second tracking frame with a matching degree greater than the matching degree threshold can be marked as a positive sample, and the second tracking frame with a matching degree not greater than the matching degree threshold can be marked as a negative sample. The training of the target detector will continue until the training is completed.

[0087] In step 103, the current image is acquired.

[0088] As an example, the current image is acquired in real time through a camera, and the current image is an image acquired in real time, and the current image and the initial image may be in the same acquisition cycle. For example, when the initial image is acquired, no operation on the photo control is received, and the image stream is continuously received by the camera until the current image is received. The current image and the initial image may also be in different acquisition cycles. For example, before the initial image is acquired and the current image is acquired, an operation on the photo control is received, which is equivalent to the initial image and the current image being in two acquisition cycles. In this case, the camera posture processing method provided in the embodiment of the present application is applied, and the current target posture of the camera is confirmed based on the first tracking frame, which can ensure that the second acquisition cycle always maintains the same acquisition stability as the first acquisition cycle.

[0089] In step 104, the current image is positioned based on the initial content by using an object detector to obtain a second tracking frame corresponding to the first tracking frame in the current image.

[0090] In some embodiments, the present application can maintain multiple camera motion states to ensure the stability of virtual object placement. Figure 3B , Figure 3B is a flow chart of the camera posture processing method provided in the embodiment of the present application. In step 104, the current image is positioned based on the initial content by the target detector to obtain a second tracking frame corresponding to the first tracking frame in the current image. Figure 3B Steps 1041 to 1042 are implemented as shown.

[0091] In step 1041, current gyroscope data when the current image is captured by the camera is obtained, and the movement speed of the camera is determined based on the current gyroscope data.

[0092] In step 1042, when the movement speed of the camera is not greater than the first speed threshold and the training process of the target detector is completed, the target detector performs positioning processing on the current image based on the initial content to obtain a second tracking frame corresponding to the first tracking frame in the current image.

[0093] The trained target detector performs positioning processing on the current image based on the initial content, and a second tracking frame corresponding to the first tracking frame can be obtained. When the camera posture is subsequently determined based on the second tracking frame and rendering is performed, the stability of the AR function can be improved, for example, the placement position of the virtual object remains unchanged, or the focus position remains unchanged.

[0094] Through the fast repositioning mode of the embodiment of the present application, the target detector is learned based on the target content of the first tracking frame obtained by user click or automatic focus. When the second tracking frame including the target content appears again during the camera movement, the current target posture of the camera can be re-determined and rendered based on the current target posture of the camera, thereby ensuring that the virtual model or the focus position remains on the first tracking frame. The camera can also be re-positioned and corrected so that the virtual model or the focus position remains on the first tracking frame during subsequent acquisition.

[0095] In some embodiments, when the camera's motion speed is greater than a first speed threshold, the camera's current posture in the world coordinate system is determined based on the camera's current gyroscope data; the camera's current posture in the world coordinate system and the camera's current position in the world coordinate system are combined to form the camera's current target posture in the world coordinate system. In high-speed motion mode, only gyroscope data is used to calculate the camera's posture (3 degrees of freedom), and the camera's current position is kept unchanged, thereby ensuring that the position of the tracking frame remains stable without drastic jumps and jitters.

[0096] As an example, see Figure 6C , Figure 6C is a logical schematic diagram of a method for processing the position and posture of a camera provided in an embodiment of the present application, Figure 6C The specific steps of rendering based on the camera posture in step 604B are shown, and the current image B is used as an example for explanation. In step 601C, it is determined whether the camera's movement speed is greater than the first speed threshold according to the rotation angular velocity in the gyroscope data. When the camera's movement speed is greater than the first speed threshold, it indicates that the camera is currently in a fast movement state, and step 602C is executed. In step 602C, the current posture of the camera is calculated using the rotation angle in the gyroscope data, and the current position of the camera is kept unchanged. In step 603C, the current posture of the camera and the current position of the camera are combined into the current target posture of the camera, and a virtual object is rendered based on the current target posture of the camera. When the camera's movement speed is not greater than the first speed threshold, it indicates that the camera is not currently in a fast movement state, and step 604C is executed. In step 604C, the current target posture of the camera is continued to be determined based on the change of the tracking frame, and a virtual object is rendered based on the current target posture of the camera.

[0097] As an example, see Fig. 6F , Fig. 6F is a logical schematic diagram of a method for processing the position and posture of a camera provided in an embodiment of the present application, Fig. 6FThe specific steps of determining the current target pose of the camera based on the change of the tracking frame are shown. The current image B is used as an example for explanation. When the movement speed of the camera is not greater than the first speed threshold, it indicates that the camera is not currently in a fast movement state. In step 601F, it is determined whether the training process of the target detector is completed. When the training process of the target detector is completed, step 602F is executed. In step 602F, the current image B is positioned by the target detector to obtain a second tracking frame corresponding to the first tracking frame in the current image B.

[0098] In some embodiments, when any one of the following conditions is met, the current image is tracked based on the third tracking frame corresponding to the first tracking frame in the previous frame of the current image to obtain a second tracking frame corresponding to the third tracking frame in the current image: the camera's movement speed is not greater than the first speed threshold, and the training process of the target detector has not yet ended; the camera's movement speed is not greater than the first speed threshold, the training process of the target detector is ended, and the overlapping area between the second tracking frame corresponding to the first tracking frame and the first tracking frame is less than the overlapping area threshold; a tracking score for the corresponding tracking process is obtained, and a tracking score threshold that is negatively correlated with the camera's movement speed is obtained; when the tracking score exceeds the tracking score threshold, the current target pose of the camera in the world coordinate system is determined based on the second tracking frame corresponding to the third tracking frame in the current image.

[0099] As an example, when the speed is lower than the first speed threshold, it indicates that the camera is in a medium-to-low speed motion state. In the medium-to-low speed motion state, adjacent front and rear images are used for visual tracking and positioning, and the current target pose of the camera when the current image is acquired is calculated in real time. As the speed decreases, the tracking score threshold of the visual algorithm tracking is increased, thereby ensuring the confidence of the tracking algorithm and reducing the jump of the virtual model or the focus position.

[0100] In some embodiments, when the tracking score does not exceed the tracking score threshold, the current posture of the camera in the world coordinate system is determined based on the current gyroscope data of the camera; the current posture of the camera in the world coordinate system and the current position of the camera in the world coordinate system are combined to form the current target posture of the camera in the world coordinate system. When the tracking score does not exceed the tracking score threshold, it is characterized that a strategy in high-speed motion mode needs to be adopted, that is, only the gyroscope data is used to calculate the camera posture (3 degrees of freedom), and the current position of the camera is kept unchanged, so as to ensure that the position of the tracking frame remains stable without drastic jumps and jitters.

[0101] As an example, see Fig. 6F , Fig. 6FThe specific steps of determining the current target pose of the camera based on the change of the tracking frame are shown. The current image B is taken as an example for explanation. In step 601F, it is determined whether the training process of the target detector is completed. When the training process of the target detector is completed, step 602F is executed. In step 602F, the current image B is positioned by the target detector to obtain the second tracking frame corresponding to the first tracking frame in the current image B. In step 603F, it is determined whether the overlap between the second tracking frame corresponding to the first tracking frame in the current image B and the first tracking frame is greater than the overlap threshold. When the overlap is not greater than the overlap threshold, step 605F is executed. In step 605F, the current target pose of the camera is calculated according to the change between the second tracking frame corresponding to the third tracking frame and the third tracking frame.

[0102] As an example, see Fig.6D , Fig.6D is a logical schematic diagram of a method for processing the position and posture of a camera provided in an embodiment of the present application, Fig.6D Specific steps of determining the current target pose of the camera based on the change of the tracking frame and rendering the virtual object based on the current target pose of the camera are shown. The current image B is taken as an example for explanation. In step 601D, it is determined whether the training process of the target detector is completed. When the training process of the target detector is not completed, step 602D is executed. In step 602D, image tracking processing based on the third tracking frame of the previous frame image C is performed on the current image B to obtain a second tracking frame corresponding to the third tracking frame in the current image B. Here, starting from obtaining the initial image (the first tracking frame of the initial image is determined by the user), The tracking frame of each frame image is tracked to obtain the tracking frame corresponding to the tracking frame in the next frame image. In step 603D, a tracking score that is negatively correlated with the movement speed of the camera is obtained. In step 604D, the relationship between the tracking score and the tracking score threshold is determined. When the tracking score of the second tracking frame corresponding to the third tracking frame in the current image B is less than the tracking score threshold, it indicates that the image tracking result is not accepted, and step 605D is executed. In step 605D, the current posture of the camera and the current position of the camera are combined into the current target posture of the camera, and the virtual object is rendered based on the current target posture of the camera.

[0103] In some embodiments, the above-mentioned determination of the current target pose of the camera in the world coordinate system based on the second tracking frame corresponding to the third tracking frame in the current image can be achieved by the following technical solutions: determining the current pose of the third tracking frame in the world coordinate system; determining the current pose of the third tracking frame in the camera coordinate system based on the second tracking frame corresponding to the third tracking frame in the current image and the third tracking frame; determining the current target pose of the camera in the world coordinate system based on the current pose of the third tracking frame in the world coordinate system and the current pose of the third tracking frame in the camera coordinate system. By obtaining the current pose of the third tracking frame in the world coordinate system and the current pose of the third tracking frame in the camera coordinate system, since the pose of the third tracking frame in the world coordinate system plays a role of coordinate system conversion on the pose of the third tracking frame in the camera coordinate system, the current target pose of the camera in the world coordinate system can be accurately determined.

[0104] As an example, based on the position and size of the second tracking frame corresponding to the third tracking frame in the current image obtained by tracking, the current target pose of the camera can be determined, that is, the camera motion is inferred based on the pose change of the same image block in the field of view, so as to calculate the current target pose of the camera. The current target pose of the camera in the world coordinate system can be calculated based on formula (1):

[0105]

[0106] Among them, T wc is the current target pose of the camera in the world coordinate system (external reference), T wa is the current position of the third tracking frame in the previous frame in the world coordinate system, T ca is the current pose of the third tracking frame in the previous frame in the camera coordinate system.

[0107] In some embodiments, the above-mentioned determination of the current position and posture of the third tracking frame in the camera coordinate system based on the second tracking frame corresponding to the third tracking frame in the current image and the third tracking frame can be implemented by the following technical solutions: determining the current depth of the third tracking frame in the camera coordinate system based on the side length of the third tracking frame, the current physical focal length of the camera and the size of the second tracking frame corresponding to the third tracking frame; determining the current position of the third tracking frame in the camera coordinate system based on the current depth of the third tracking frame in the camera coordinate system, the internal parameters of the camera and the current position of the second tracking frame corresponding to the third tracking frame in the world coordinate system; and combining the current position of the third tracking frame in the camera coordinate system and the current posture of the camera in the camera coordinate system to form the current position and posture of the third tracking frame in the camera coordinate system. Combining the current depth of the third tracking frame in the camera coordinate system and the current position of the second tracking frame corresponding to the third tracking frame in the world coordinate system can improve the accuracy of determining the current position of the third tracking frame in the camera coordinate system, thereby improving the accuracy of the current position and posture of the third tracking frame in the camera coordinate system.

[0108] As an example, the current position of the third tracking frame in the previous frame image in the camera coordinate system can be calculated based on formula (2)-formula (4):

[0109] D A =A w *K(x) / B w (2);

[0110] Among them, D A is the current depth of the camera at the third tracking frame in the previous frame, and K(x) refers to the physical focal length of the camera f x , A w is the width of the third tracking box in the previous frame, B w is the size of the second tracking box corresponding to the third tracking box in the current image.

[0111] t ca =D A *K -1 *B (3);

[0112] Among them, t ca is the current position of the third tracking frame in the previous frame image in the camera coordinate system, K is the camera intrinsic parameter, and B is the position of the second tracking frame corresponding to the third tracking frame in the current image.

[0113] T ca =(R ca ,t ca ) (4);

[0114] Among them, R cais the current pose of the camera obtained from the gyroscope, which can be used as the current pose of the third tracking frame in the previous frame image. ca is the current pose of the third tracking frame in the previous frame in the camera coordinate system.

[0115] In step 105, based on the second tracking frame corresponding to the first tracking frame in the current image, the current target pose of the camera in the world coordinate system is determined.

[0116] In some embodiments, before determining the current target pose of the camera in the world coordinate system based on the second tracking frame corresponding to the first tracking frame in the current image in step 105, when the second tracking frame corresponding to the first tracking frame is located in the boundary area of ​​the current image, a fourth tracking frame of the current image is obtained; wherein the fourth tracking frame is adjacent to the second tracking frame and is closer to the center position of the current image than the second tracking frame; and the fourth tracking frame is determined as the second tracking frame. By correcting the position quality of the second tracking frame through the fourth tracking frame, thereby avoiding the second tracking frame being in the boundary area, the stability of the AR function can be maintained in the subsequent image tracking process, for example, the position of the virtual object is kept unchanged or the focus is kept unchanged.

[0117] As an example, see Fig. 6E , Fig. 6E is a logical schematic diagram of a method for processing the position and posture of a camera provided in an embodiment of the present application, Fig. 6E The processing flow after successful repositioning is shown. The current image B is used as an example for explanation. In step 601E, it is determined whether the second tracking frame corresponding to the first tracking frame is located in the boundary area of ​​the current image. When it is located in the boundary area, step 602E is executed to obtain the fourth tracking frame of the current image. The fourth tracking frame is adjacent to the second tracking frame and is closer to the center of the current image than the second tracking frame. In step 603E, the fourth tracking frame is determined as the second tracking frame and the second tracking frame is displayed in the current image.

[0118] In some embodiments, see Figure 3C , Figure 3C is a flow chart of the camera posture processing method provided in an embodiment of the present application. In step 105, based on the second tracking frame corresponding to the first tracking frame in the current image, the current target posture of the camera in the world coordinate system is determined. Figure 3C Steps 1051 to 1053 are implemented as shown.

[0119] In step 1051, the current position and posture of the first tracking frame in the world coordinate system is determined;

[0120] In step 1052, based on the second tracking frame corresponding to the first tracking frame and the first tracking frame, a current position of the first tracking frame in the camera coordinate system is determined;

[0121] In step 1053 , based on the current pose of the first tracking frame in the world coordinate system and the current pose of the first tracking frame in the camera coordinate system, the current target pose of the camera in the world coordinate system is determined.

[0122] As an example, the current target pose of the camera can be calculated by formula (5):

[0123]

[0124] Among them, T wc is the current target pose of the camera in the world coordinate system (external reference), is the current position of the first tracking box in the initial image in the world coordinate system, is the current pose of the first tracking box of the initial image in the camera coordinate system.

[0125] The current pose of the first tracking frame in the world coordinate system and the current pose of the first tracking frame in the camera coordinate system are obtained. Since the pose of the first tracking frame in the world coordinate system plays a role of coordinate system conversion for the pose of the first tracking frame in the camera coordinate system, the current target pose of the camera in the world coordinate system can be accurately determined.

[0126] In some embodiments, based on the third tracking frame corresponding to the first tracking frame in the previous frame of the current image, the current image is tracked to obtain a second tracking frame corresponding to the third tracking frame in the current image; when the position difference between the second tracking frame corresponding to the third tracking frame and the second tracking frame corresponding to the first tracking frame is less than the position difference threshold, the current position of the camera in the world coordinate system is smoothed and corrected, so that the camera moves from the current position to the current target position corresponding to the current target posture in the world coordinate system; when the position difference between the second tracking frame corresponding to the third tracking frame and the second tracking frame corresponding to the first tracking frame is not less than the position difference threshold, the camera is updated from the current position to the current target position corresponding to the current target posture in the world coordinate system. In order to prevent the virtual model or focus position from jumping after repositioning, the current posture of the camera can be smoothed so that the virtual model can slowly move to the position of the first tracking frame. Different position movement strategies are adopted based on different position gaps. When the position gap is less than the position gap threshold, a smooth correction method is used to move the position, thereby improving the visual stability of the AR function. When the position gap is not less than the position gap threshold, the position is directly updated, which can improve the human-computer interaction efficiency of the AR function.

[0127] As an example, the current position of the camera is corrected based on the current target position in the current target pose of the camera. For corrections within the threshold range, for example, if the position difference between the second tracking frame corresponding to the third tracking frame and the second tracking frame corresponding to the first tracking frame is less than the position difference threshold, a smoothing method is used to smooth the camera position. Otherwise, the current camera position is directly updated, and the camera is directly moved from the current position to the current target position corresponding to the current target pose in the world coordinate system. A variety of smoothing strategies can be adopted, such as sliding window averaging, distance weighted averaging, frame rate weighted averaging, etc. For sliding window averaging, formula (6) can be used:

[0128]

[0129] Among them, n is the sliding window size, t wc is the current target position corresponding to the current target pose of the camera.

[0130] In some embodiments, the above-mentioned determination of the current position and posture of the first tracking frame in the camera coordinate system based on the second tracking frame corresponding to the first tracking frame and the first tracking frame can be implemented by the following technical solutions: determining the current depth of the first tracking frame in the camera coordinate system based on the side length of the first tracking frame, the current physical focal length of the camera and the size of the second tracking frame corresponding to the first tracking frame; determining the current position of the first tracking frame in the camera coordinate system based on the current depth of the first tracking frame in the camera coordinate system, the internal parameters of the camera and the current position of the second tracking frame corresponding to the first tracking frame in the world coordinate system; and combining the current position of the first tracking frame in the camera coordinate system and the current posture of the first tracking frame in the camera coordinate system to form the current position and posture of the first tracking frame in the camera coordinate system. Combining the current depth of the first tracking frame in the camera coordinate system and the current position of the second tracking frame corresponding to the first tracking frame in the world coordinate system can improve the accuracy of determining the current position of the first tracking frame in the camera coordinate system, thereby improving the accuracy of the current position and posture of the first tracking frame in the camera coordinate system.

[0131] As an example, the current pose of the first tracking frame in the camera coordinate system can be calculated based on formula (7):

[0132]

[0133] in, is the current pose of the camera obtained from the gyroscope, which can be used as the current pose of the first tracking box of the initial image, is the current pose of the first tracking frame of the initial image in the camera coordinate system, is the current position of the first tracking box of the initial image in the camera coordinate system.

[0134] As an example, the current depth of the first tracking frame in the camera coordinate system can be calculated based on formula (8):

[0135]

[0136] in, is the current depth of the camera at the first tracking frame of the initial image, and K(x) refers to the physical focal length of the camera f x , is the width of the first tracking box of the initial image, B w is the size of the second tracking box corresponding to the first tracking box in the current image.

[0137] As an example, the current position of the first tracking frame of the initial image in the camera coordinate system can be calculated based on formula (9):

[0138]

[0139] in, is the current position of the first tracking frame of the initial image in the camera coordinate system, K is the camera intrinsic parameter, and B is the position of the second tracking frame corresponding to the first tracking frame in the current image.

[0140] In some embodiments, the above-mentioned determination of the current position and posture of the first tracking frame in the world coordinate system can be achieved by the following technical solutions: determining the initial position and posture of the first tracking frame in the camera coordinate system; determining the initial position and posture of the camera in the world coordinate system; determining the current position and posture of the first tracking frame in the world coordinate system based on the initial position and posture of the camera in the world coordinate system and the initial position and posture of the first tracking frame in the camera coordinate system. Since there is a conversion relationship between the camera coordinate system and the world coordinate system, the current position and posture of the first tracking frame in the world coordinate system is not only related to the initial position and posture of the first tracking frame in the camera coordinate system, but also related to the initial position and posture of the camera in the world coordinate system. Combining the initial position and posture of the camera in the world coordinate system and the initial position and posture of the first tracking frame in the camera coordinate system, the current position and posture of the first tracking frame in the world coordinate system is determined, which can improve the accuracy of determining the current position and posture of the first tracking frame in the world coordinate system.

[0141] As an example, the initial pose of the first tracking frame in the world coordinate system is calculated based on formula (10):

[0142]

[0143] Among them, T wa0 is the initial pose of the first tracking frame in the world coordinate system, is the initial pose of the camera in the world coordinate system, is the initial pose of the first tracking box in the camera coordinate system.

[0144] In some embodiments, the above-mentioned determination of the initial posture of the camera in the world coordinate system can be achieved through the following technical solutions: determining the initial position of the camera in the world coordinate system; obtaining the initial gyroscope data of the camera corresponding to the first tracking frame, and determining the initial posture of the camera corresponding to the initial gyroscope data; and combining the initial position of the camera in the world coordinate system and the initial posture of the camera to form the initial posture of the camera in the world coordinate system.

[0145] As an example, the world coordinate system is established with the click point as the coordinate origin, and the gyroscope data is used to determine the initial attitude angle of the camera when the initial image is collected. Use trigonometric relationships to calculate the initial position of the camera relative to the click point in the world coordinate system when acquiring the initial image The initial camera posture angle And the camera position The initial pose of the camera in the world coordinate system

[0146] In some embodiments, the above-mentioned determination of the initial position and posture of the first tracking frame in the camera coordinate system can be achieved by the following technical solutions: based on the horizontal plane normal vector, the internal parameters of the camera and the initial position of the first tracking frame in the world coordinate system, the initial depth of the first tracking frame in the camera coordinate system is determined; based on the initial depth of the first tracking frame in the camera coordinate system, the internal parameters of the camera and the initial position of the first tracking frame in the world coordinate system, the initial position and posture of the first tracking frame in the camera coordinate system is determined. Combining the initial depth of the first tracking frame in the camera coordinate system and the initial position of the first tracking frame in the world coordinate system can improve the accuracy of determining the initial position of the first tracking frame in the camera coordinate system, thereby improving the accuracy of the initial position and posture of the first tracking frame in the camera coordinate system.

[0147] As an example, the initial pose of the first tracking frame in the camera coordinate system is calculated based on formula (11):

[0148]

[0149] in, is the initial pose of the first tracking frame in the camera coordinate system, D i Indicates the depth of the click point in the camera coordinate system, the camera intrinsic parameter is K, and the target position is P.

[0150] As an example, the depth of the target position in the camera coordinate system is calculated based on formula (12):

[0151] D i =-1* / (N p 23*K -1 *P) (12);

[0152] Among them, D i Indicates the depth of the target position in the camera coordinate system, the target position is P, the camera intrinsic parameter is K, and the horizontal plane normal vector is N p (Default gravity is in the opposite direction).

[0153] By learning the texture of the image area block where the user initially placed the point, when the initial image area block appears again in the camera image, the camera position can be recalculated and gradually restored to the camera position when the user initially clicked through a smoothing algorithm, so that the AR model remains in place, thereby solving the problem of accumulated drift of the AR model. At the same time, by maintaining multiple camera motion states, switching the threshold parameters of the tracking algorithm, and combining gyroscope data, the positioning algorithm is made more robust, further reducing the problem of accumulated drift of the AR model.

[0154] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0155] The camera posture processing method provided in the embodiment of the present application can be applied to a short video APP. The short video APP will provide an AR-based acquisition function, such as placing a virtual model on the acquired original scene in response to a user's click operation. Figures 4A-4B , Figures 4A-4B It is a display interface diagram of the camera posture processing method provided in an embodiment of the present application. In response to a user's trigger operation on a target position, a virtual model is placed at the target position. The processor of the terminal receives an image stream captured by the camera (the camera can be set at the terminal or can be independent of the terminal). In response to a user's click operation on the target position in the human-computer interaction interface of the terminal, the terminal renders a virtual model based on the target position, determines a first tracking frame based on the target position and an initial image captured when the first tracking frame is determined, and the terminal sends an initial image including the first tracking frame to a server. The server trains the image based on the initial content of the first tracking frame of the initial image. The target detector that recognizes the initial content, while training, the processor of the terminal is still receiving the image stream captured by the camera, the terminal obtains the current image currently captured by the camera in real time, and also sends the current image to the server, the server performs positioning processing on the current image based on the initial content through the target detector, obtains the second tracking frame corresponding to the first tracking frame in the current image, and returns the second tracking frame in the current image to the terminal, the terminal determines the current target pose of the camera in the world coordinate system, and then the terminal corrects the position of the camera based on the current target pose of the camera in the world coordinate system, and renders the virtual model based on the corrected camera position.

[0156] The camera posture processing method provided in the embodiment of the present application learns the image texture in the first tracking frame of the user's initial placement point. When the image in the first tracking frame appears again in the current image of the camera, the current target posture of the camera can be recalculated, and rendering is performed based on the current target posture. Through a smoothing algorithm, the camera is gradually restored to the position when the user initially clicked, so that the virtual model can remain in place and remain stable in place during subsequent acquisition processes, thereby solving the problem of cumulative drift of the virtual model. In addition, by maintaining multiple camera motion states, switching the threshold parameters of the tracking algorithm, and combining gyroscope data, the positioning algorithm is made more robust, further reducing the problem of cumulative drift of the virtual model.

[0157] In some embodiments, see Figure 5 , Figure 5 It is an algorithm structure diagram of the camera posture processing method provided in the embodiment of the present application. The camera posture processing algorithm provided in the embodiment of the present application includes the following modules: a sensor input module, a camera 6-degree-of-freedom posture calculation module and a rendering module, wherein the camera 6-degree-of-freedom posture calculation module includes a repositioning module, a posture calculation module, a tracking module and a position calculation module, the posture calculation module is used to obtain the 3-degree-of-freedom camera posture using gyroscope data, the tracking module is used to track the image frame of the image, the position calculation module is used to calculate the 3-degree-of-freedom position of the camera using the result of the tracking module, and the repositioning module is used to detect whether the camera moves to the initial position, thereby correcting the current camera position.

[0158] See also Fig. 6A , Fig. 6AIt is a logical schematic diagram of the camera posture processing method provided in an embodiment of the present application. The camera collects an image stream, and the following processing is performed for each current image in each image stream (the current image A is taken as an example for explanation): in step 601A, the current image A and the gyroscope data corresponding to the current image A are obtained, wherein the gyroscope data corresponding to the current image A is the gyroscope data of the nearest adjacent timestamp of the current image. In step 602A, it is determined whether an operation of the user clicking on the target position to place the virtual model is received. When the operation of the user clicking on the target position to place the virtual model is received, step 603A is executed. In step 603A, initialization processing is performed based on the initial image (current image A) collected when the virtual model is placed. After the initialization is completed, step 604A is executed. In step 604A, based on the current image A (initial image) and the first tracking frame determined based on the target position, a target detector for identifying the content of the first tracking frame in the current image A (initial image) is trained. The training process of the target detector can be carried out in a variety of ways, for example, the kernel correlation filtering method. The training goal is to train the target detector for tracking frame detection. The target detector can detect the tracking frame including the content of the first tracking frame. When training the target detector, the area of ​​the first tracking frame is usually selected as the positive sample, and the surrounding area of ​​the first tracking frame is selected as the negative sample. During the forward propagation, the target detector is used to detect the next frame image to obtain the second tracking frame corresponding to the first tracking frame in the next frame image, and then the area of ​​the second tracking frame is used to update the training sample set to continue training the target detector, wherein the detected second tracking frame can be marked as a positive sample or a negative sample based on the matching degree with the first tracking frame. The training of the target detector will continue until the training is completed.

[0159] In some embodiments, the initialization process is specifically described as follows: the field of view scene is an infinite horizontal plane, the world coordinate system is established with the click point as the coordinate origin, the camera height is 1, and the gyroscope data is used to determine the initial attitude angle of the camera when the initial image is collected. Use trigonometric relationships to calculate the initial position of the camera relative to the click point in the world coordinate system when acquiring the initial image The initial camera posture angle And the camera position The initial pose of the camera in the world coordinate system Based on formula (13)-formula (15), the initial position of the first tracking frame corresponding to the click point P in the world coordinate system is calculated The first tracking frame is the tracking frame of the initial image set with the click point as the center, and its width is The default setting is rectangular blocks of 1 / 2 the image resolution:

[0160] D i=-1* / (N p 26*K -1 *P) (13);

[0161]

[0162]

[0163] Among them, D i Indicates the depth of the click point in the camera coordinate system. Represents the position of the click point in the camera coordinate system. The pixel position of the click point is P, the camera intrinsic parameter is K, and the horizontal plane normal vector is N p (Default gravity is in the opposite direction).

[0164] See also Figure 6B , Figure 6B It is a logical schematic diagram of the camera posture processing method provided in an embodiment of the present application. The following processing is performed for each current image in each image stream (the current image B is taken as an example for explanation): in step 601B, the current image B and the gyroscope data corresponding to the current image B are obtained, wherein the gyroscope data corresponding to the current image B is the gyroscope data of the nearest adjacent timestamp of the current image. In step 602B, it is determined whether an operation of the user clicking on the target position to place the virtual model is received. When the operation of the user clicking on the target position to place the virtual model is not received, step 603B is executed. In step 603B, it is determined whether the virtual model has been placed. When the virtual model has not been placed, the process is terminated. When the virtual model has been placed, step 604B is executed. In step 604B, rendering based on the camera posture is performed.

[0165] See also Figure 6C , Figure 6C The specific steps of rendering based on the camera posture in step 604B are shown, and the current image B is used as an example for explanation. In step 601C, it is determined whether the camera's movement speed is greater than the first speed threshold according to the rotation angular velocity in the gyroscope data. When the camera's movement speed is greater than the first speed threshold, it indicates that the camera is currently in a fast movement state, and step 602C is executed. In step 602C, the current posture of the camera is calculated using the rotation angle in the gyroscope data, and the current position of the camera is kept unchanged. In step 603C, the current posture of the camera and the current position of the camera are combined into the current target posture of the camera, and a virtual object is rendered based on the current target posture of the camera. When the camera's movement speed is not greater than the first speed threshold, it indicates that the camera is not currently in a fast movement state, and step 604C is executed. In step 604C, the current target posture of the camera is continued to be determined based on the change of the tracking frame, and a virtual object is rendered based on the current target posture of the camera.

[0166] See also Fig.6D , Fig.6D Specific steps of determining the current target pose of the camera based on the change of the tracking frame and rendering the virtual object based on the current target pose of the camera are shown. The current image B is taken as an example for explanation. In step 601D, it is determined whether the training process of the target detector is completed. When the training process of the target detector is not completed, step 602D is executed. In step 602D, image tracking processing based on the third tracking frame of the previous frame image C is performed on the current image B to obtain a second tracking frame corresponding to the third tracking frame in the current image B. Here, starting from obtaining the initial image (the first tracking frame of the initial image is determined by the user), The tracking frame of each frame image is tracked to obtain the tracking frame corresponding to the tracking frame in the next frame image. In step 603D, a tracking score that is negatively correlated with the camera's motion speed is obtained. In step 604D, the magnitude relationship between the tracking score and the tracking score threshold is determined. When the tracking score of the second tracking frame corresponding to the third tracking frame in the current image B is less than the tracking score threshold, it indicates that the image tracking result is not accepted, and step 605D is executed. In step 605D, the current posture of the camera and the current position of the camera are combined into the current target posture of the camera, and the virtual object is rendered based on the current target posture of the camera. When the tracking score of the second tracking frame corresponding to the third tracking frame in the current image B is not less than the tracking score threshold, it indicates that the image tracking result is accepted, and step 606D is executed. In step 606D, the current target posture of the camera is calculated according to the change of the tracking frame, and the virtual object is rendered based on the current target posture of the camera.

[0167] In some embodiments, an optical flow algorithm or other algorithms can be used to track image blocks. Taking the Lucas-Kanade optical flow tracking method as an example, the image corner points of the third tracking frame in the previous frame image are tracked to obtain new image corner points in the current image, and the position and size of the image corner points are determined, thereby obtaining the second tracking frame in the current image corresponding to the third tracking frame in the previous frame image.

[0168] In some embodiments, the three-dimensional pose is restored based on the position and size of the second tracking frame corresponding to the third tracking frame in the current image obtained by tracking, that is, the camera motion is inferred based on the pose change of the same image block in the field of view, so as to calculate the current target pose of the camera, see formula (16)-formula (19):

[0169] D A =A w *K(x) / B w (16);

[0170] Among them, D A is the current depth of the camera at the third tracking frame in the previous frame, and K(x) refers to the physical focal length of the camera f x, A w is the width of the third tracking box in the previous frame, B w is the size of the second tracking box corresponding to the third tracking box in the current image.

[0171] t ca =D A *K -1 *B (17);

[0172] Among them, t ca is the current position of the third tracking frame in the previous frame image in the camera coordinate system, K is the camera intrinsic parameter, and B is the position of the second tracking frame corresponding to the third tracking frame in the current image.

[0173] T ca =(R ca ,t ca ) (18);

[0174] Among them, R ca is the current pose of the camera obtained from the gyroscope, which can be used as the current pose of the third tracking frame in the previous frame image. ca is the current pose of the third tracking frame in the previous frame in the camera coordinate system.

[0175]

[0176] Among them, T wc is the current target pose of the camera in the world coordinate system (external reference), T wa It is the current position of the third tracking box in the previous frame in the world coordinate system.

[0177] In some embodiments, see Fig. 6F , Fig. 6F is a logical schematic diagram of a method for processing the position and posture of a camera provided in an embodiment of the present application, Fig. 6FThe specific steps of determining the current target pose of the camera based on the change of the tracking frame are shown. The current image B is used as an example for explanation. In step 601F, it is determined whether the training process of the target detector is completed. When the training process of the target detector is completed, step 602F is executed. In step 602F, the current image B is positioned by the target detector to obtain the second tracking frame corresponding to the first tracking frame in the current image B. In step 603F, it is determined whether the overlap between the second tracking frame corresponding to the first tracking frame in the current image B and the first tracking frame is greater than the overlap threshold. When the overlap is greater than the overlap threshold, step 604F is executed. In step 604F, the current target pose of the camera is calculated according to the change between the second tracking frame corresponding to the first tracking frame and the first tracking frame. When the overlap is not greater than the overlap threshold, step 605F is executed. In step 605F, the current target pose of the camera is calculated according to the change between the second tracking frame corresponding to the third tracking frame and the third tracking frame.

[0178] In some embodiments, the target detector is used to detect whether the user's camera moves back to the vicinity of the click point, so that the current target pose can be calculated. The target detector performs first tracking frame matching to determine whether the first tracking frame clicked by the user is found. If a second tracking frame matching the first tracking frame is identified (the overlap is greater than the overlap threshold), the relocation is considered successful, and the current target pose of the camera can be calculated. The current target pose of the camera can be calculated by formula (20)-formula (23):

[0179]

[0180] in, is the current depth of the camera at the first tracking frame of the initial image, and K(x) refers to the physical focal length of the camera f x , is the width of the first tracking box of the initial image, B w is the size of the second tracking box corresponding to the first tracking box in the current image.

[0181]

[0182] in, is the current position of the first tracking frame of the initial image in the camera coordinate system, K is the camera intrinsic parameter, and B is the position of the second tracking frame corresponding to the first tracking frame in the current image.

[0183]

[0184] in, is the current pose of the camera obtained from the gyroscope, which can be used as the current pose of the first tracking box of the initial image, is the current pose of the first tracking box of the initial image in the camera coordinate system.

[0185]

[0186] Among them, T wc is the current target pose of the camera in the world coordinate system (external reference), is the current pose of the first tracking box in the initial image in the world coordinate system.

[0187] In some embodiments, see Figure 6G , Figure 6G is a logical schematic diagram of a method for processing the position and posture of a camera provided in an embodiment of the present application, Figure 6G The figure shows the processing flow after the current target pose of the camera is determined. The current image B is used as an example for explanation. In step 601G, the current target pose of the camera is determined. In step 602G, the virtual model is rendered based on the current target pose of the camera. In step 603G, the position difference between the second tracking frame corresponding to the first tracking frame obtained by repositioning and the second tracking frame corresponding to the third tracking frame obtained based on tracking is determined. When the position difference is greater than the difference threshold, step 604G is executed to directly move the camera from the current position in the world coordinate system to the current target position corresponding to the current target pose. When the position difference is not greater than the difference threshold, step 605G is executed to perform smoothing correction processing on the current position of the camera in the world coordinate system, so that the camera is at the current target position corresponding to the current target pose in the world coordinate system.

[0188] In some embodiments, the current position of the camera is corrected based on the current target position in the current target pose of the camera. For corrections within the threshold range, for example, when the position difference is not greater than the difference threshold, a smoothing method is used to smooth the camera position. Otherwise, the current camera position is directly updated, and the camera is directly moved from the current position to the current target position corresponding to the current target pose in the world coordinate system. A variety of smoothing strategies can be adopted, such as sliding window averaging, distance weighted averaging, frame rate weighted averaging, etc. For sliding window averaging, formula (24) can be used:

[0189]

[0190] Among them, n is the sliding window size, t wc is the current target position corresponding to the current target pose of the camera.

[0191] In some embodiments, see Fig. 6E , Fig. 6EThe processing flow after successful repositioning is shown. The current image B is used as an example for explanation. In step 601E, it is determined whether the second tracking frame corresponding to the first tracking frame is located in the boundary area of ​​the current image. When it is located in the boundary area, step 602E is executed to obtain the fourth tracking frame of the current image. The fourth tracking frame is adjacent to the second tracking frame and is closer to the center of the current image than the second tracking frame. In step 603E, the fourth tracking frame is determined as the second tracking frame and the second tracking frame is displayed in the current image.

[0192] The camera posture processing method provided in the embodiment of the present application is aimed at augmented reality scenarios, and proposes a camera positioning solution based on rapid repositioning to calculate the current target posture of the camera, including the camera's 3-degree-of-freedom spatial position and 3-degree-of-freedom posture (orientation), collectively referred to as the camera's 6-degree-of-freedom posture, to meet the camera positioning requirements in augmented reality functions.

[0193] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0194] The following is a description of an exemplary structure of a camera posture processing device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules in the camera posture processing device 455 stored in the memory 450 may include: an initial module 4551, used to obtain an initial image including a first tracking frame, wherein the initial image is acquired when the camera is first positioned to the first tracking frame; a training module 4552, used to train a target detector for identifying the initial content based on the initial content of the first tracking frame of the initial image; a current module 4553, used to obtain a current image, wherein the current image is acquired in real time by the camera; a repositioning module 4554, used to perform positioning processing on the current image based on the initial content through the target detector, and obtain a second tracking frame corresponding to the first tracking frame in the current image; a determination module 4555, used to determine the current target posture of the camera in the world coordinate system based on the second tracking frame corresponding to the first tracking frame in the current image.

[0195] In some embodiments, before acquiring the initial image including the first tracking frame, the initial module 4551 is further used to: in response to a click operation on the target position, perform any one of the following processes: determine a tracking frame with the target position as the center point and a set size as the first tracking frame; determine a tracking frame with the target position as a set corner point and a set size as the first tracking frame.

[0196] In some embodiments, the training module 4552 is also used to: when the test result of the target detector is less than the test result threshold and the cumulative number of batch training of the target detector does not exceed the number threshold, perform the following processing: batch training the target detector based on the training sample set, wherein the training sample set includes training samples corresponding to the initial content; add the recognition results obtained based on the training sample set during the batch training to the training sample set to update the training sample set.

[0197] In some embodiments, the determination module 4555 is also used to: determine the current pose of the first tracking frame in the world coordinate system; determine the current pose of the first tracking frame in the camera coordinate system based on the second tracking frame corresponding to the first tracking frame and the first tracking frame; determine the current target pose of the camera in the world coordinate system based on the current pose of the first tracking frame in the world coordinate system and the current pose of the first tracking frame in the camera coordinate system.

[0198] In some embodiments, the determination module 4555 is also used to: perform tracking processing on the current image based on the third tracking frame corresponding to the first tracking frame in the previous frame image of the current image to obtain a second tracking frame corresponding to the third tracking frame in the current image; when the position difference between the second tracking frame corresponding to the third tracking frame and the second tracking frame corresponding to the first tracking frame is less than a position difference threshold, perform smoothing correction processing on the current position of the camera in the world coordinate system so that the camera moves from the current position to the current target position corresponding to the current target posture in the world coordinate system; when the position difference between the second tracking frame corresponding to the third tracking frame and the second tracking frame corresponding to the first tracking frame is not less than the position difference threshold, update the camera from the current position to the current target position corresponding to the current target posture in the world coordinate system.

[0199] In some embodiments, the determination module 4555 is further used to: determine the current depth of the first tracking frame in the camera coordinate system based on the side length of the first tracking frame, the current physical focal length of the camera, and the size of the second tracking frame corresponding to the first tracking frame; determine the current posture of the first tracking frame in the camera coordinate system based on the current depth of the first tracking frame in the camera coordinate system, internal parameters of the camera, and the current position of the second tracking frame corresponding to the first tracking frame in the world coordinate system; and combine the current position of the first tracking frame in the camera coordinate system and the current posture of the first tracking frame in the camera coordinate system to form the current posture of the first tracking frame in the camera coordinate system.

[0200] In some embodiments, the determination module 4555 is also used to: determine the initial pose of the first tracking frame in the camera coordinate system; determine the initial pose of the camera in the world coordinate system; and determine the current pose of the first tracking frame in the world coordinate system based on the initial pose of the camera in the world coordinate system and the initial pose of the first tracking frame in the camera coordinate system.

[0201] In some embodiments, the determination module 4555 is also used to: determine the initial position of the camera in the world coordinate system; obtain the initial gyroscope data of the camera corresponding to the first tracking frame, and determine the initial posture of the camera corresponding to the initial gyroscope data; and combine the initial position of the camera in the world coordinate system and the initial posture of the camera into the initial posture of the camera in the world coordinate system.

[0202] In some embodiments, the determination module 4555 is also used to: determine the initial depth of the first tracking frame in the camera coordinate system based on the horizontal plane normal vector, the internal parameters of the camera, and the initial position of the first tracking frame in the world coordinate system; determine the initial pose of the first tracking frame in the camera coordinate system based on the initial depth of the first tracking frame in the camera coordinate system, the internal parameters of the camera, and the initial position of the first tracking frame in the world coordinate system.

[0203] In some embodiments, the repositioning module 4554 is also used to: obtain current gyroscope data when the current image is captured by the camera; when the movement speed of the camera is not greater than the first speed threshold and the training process of the target detector is completed, the current image is positioned based on the initial content through the target detector to obtain a second tracking frame corresponding to the first tracking frame in the current image.

[0204] In some embodiments, the determination module 4555 is also used to: when the movement speed of the camera is greater than a first speed threshold, determine the current posture of the camera in the world coordinate system based on the current gyroscope data of the camera; keep the position of the camera in the world coordinate system unchanged, and combine the current posture of the camera in the world coordinate system and the current position of the camera in the world coordinate system to form the current target posture of the camera in the world coordinate system.

[0205] In some embodiments, the determination module 4555 is further used to: when any one of the following conditions is met, based on the third tracking frame corresponding to the first tracking frame in the previous frame image of the current image, track the current image to obtain a second tracking frame corresponding to the third tracking frame in the current image: the camera's movement speed is not greater than the first speed threshold, and the training process of the target detector has not yet ended; the camera's movement speed is not greater than the first speed threshold, the training process of the target detector is ended, and the overlapping area between the second tracking frame corresponding to the first tracking frame and the first tracking frame is less than the overlapping area threshold; obtain a tracking score for the corresponding tracking processing, and obtain a tracking score threshold that is negatively correlated with the camera's movement speed; when the tracking score exceeds the tracking score threshold, determine the current target pose of the camera in the world coordinate system based on the second tracking frame corresponding to the third tracking frame in the current image.

[0206] In some embodiments, the determination module 4555 is also used to: when the tracking score does not exceed the tracking score threshold, determine the current posture of the camera in the world coordinate system based on the current gyroscope data of the camera; keep the position of the camera in the world coordinate system unchanged, and combine the current posture of the camera in the world coordinate system and the current position of the camera in the world coordinate system to form the current target posture of the camera in the world coordinate system.

[0207] In some embodiments, the determination module 4555 is also used to: determine the current pose of the third tracking frame in the world coordinate system; determine the current pose of the third tracking frame in the camera coordinate system based on the second tracking frame corresponding to the third tracking frame in the current image, and the third tracking frame; determine the current target pose of the camera in the world coordinate system based on the current pose of the third tracking frame in the world coordinate system and the current pose of the third tracking frame in the camera coordinate system.

[0208] In some embodiments, determination module 4555 is further used to: determine the current depth of the third tracking frame in the camera coordinate system based on the side length of the third tracking frame, the current physical focal length of the camera, and the size of the second tracking frame corresponding to the third tracking frame; determine the current posture of the third tracking frame in the camera coordinate system based on the current depth of the third tracking frame in the camera coordinate system, internal parameters of the camera, and the current position of the second tracking frame corresponding to the third tracking frame in the world coordinate system; and combine the current position of the third tracking frame in the camera coordinate system and the current posture of the camera in the camera coordinate system to form the current posture of the third tracking frame in the camera coordinate system.

[0209] In some embodiments, before determining the current target pose of the camera in the world coordinate system based on the second tracking frame corresponding to the first tracking frame in the current image, the determination module 4555 is also used to: when the second tracking frame corresponding to the first tracking frame is located in the boundary area of ​​the current image, obtain a fourth tracking frame of the current image; wherein the fourth tracking frame is adjacent to the second tracking frame and is closer to the center position of the current image than the second tracking frame; and determine the fourth tracking frame as the second tracking frame.

[0210] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the camera posture processing method described in the embodiment of the present application.

[0211] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the camera posture processing method provided by the present application embodiment, for example, Figures 3A-3C The camera pose processing method is shown.

[0212] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0213] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0214] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0215] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0216] In summary, through the embodiment of the present application, by learning the initial content of the first tracking frame, a target detector for identifying the initial content is trained, so that the initial content can be identified in the current image collected in real time, so as to locate the second tracking frame corresponding to the first tracking frame, and then determine the current target pose of the camera in the world coordinate system according to the second tracking frame. Since the second tracking frame is obtained based on the initial content positioning, the current target pose determined based on the second tracking frame matches the camera pose when the initial image is collected, thereby improving the accuracy of the camera pose repositioning.

[0217] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A camera posture processing method, characterized in that: The method comprises: Acquire an initial image including a first tracking frame, wherein the initial image is acquired by the camera when the first tracking frame is first positioned; Based on the initial content of the first tracking box of the initial image, training an object detector for recognizing the initial content; Acquire a current image, wherein the current image is acquired in real time by the camera; Acquire current gyroscope data when the current image is captured by the camera, and determine the movement speed of the camera based on the current gyroscope data; When the movement speed of the camera is not greater than a first speed threshold and the training process of the object detector is completed, performing positioning processing on the current image based on the initial content by the object detector to obtain a second tracking frame in the current image corresponding to the first tracking frame; When an overlapping area between the second tracking frame of the first tracking frame and the first tracking frame is less than an overlapping area threshold, tracking the current image based on a third tracking frame corresponding to the first tracking frame in a previous frame of the current image to obtain a second tracking frame corresponding to the third tracking frame in the current image; Obtaining a tracking score corresponding to the tracking process, and obtaining a tracking score threshold that is negatively correlated with the motion speed of the camera; When the tracking score exceeds the tracking score threshold, determining a current target pose of the camera in a world coordinate system based on a second tracking frame corresponding to the third tracking frame in the current image, wherein the current target pose is a camera pose with six degrees of freedom; When the tracking score does not exceed the tracking score threshold, based on the current gyroscope data of the camera, determine the current posture of the camera in the world coordinate system, wherein the current posture is a camera posture with three degrees of freedom; and combine the current posture of the camera in the world coordinate system and the current position of the camera in the world coordinate system to form the current target posture of the camera in the world coordinate system.

2. The method according to claim 1, characterized in that: Before acquiring the initial image including the first tracking frame, the method further includes: In response to a click operation on a target location, any one of the following processes is performed: Determine a tracking frame with the target position as the center point and a set size as the first tracking frame; A tracking frame with the target position as a set corner point and a set size is determined as the first tracking frame.

3. The method according to claim 1, characterized in that: The step of training a target detector for identifying the initial content based on the initial content of the first tracking frame of the initial image comprises: When the test result of the target detector is less than the test result threshold, and the cumulative number of batch training of the target detector does not exceed the number threshold, the following processing is performed: Performing batch training on the object detector based on a training sample set, wherein the training sample set includes training samples corresponding to the initial content; The method further comprises: The recognition results obtained based on the training sample set during the batch training are added to the training sample set to update the training sample set.

4. The method according to claim 1, characterized in that: After determining the current target pose of the camera in the world coordinate system, the method further includes: Based on a third tracking frame corresponding to the first tracking frame in a previous frame of the current image, tracking the current image is performed to obtain a second tracking frame corresponding to the third tracking frame in the current image; When the position difference between the second tracking frame corresponding to the third tracking frame and the second tracking frame corresponding to the first tracking frame is less than a position difference threshold, a smooth correction process is performed on the current position of the camera in the world coordinate system so that the camera moves from the current position to the current target position corresponding to the current target posture in the world coordinate system; When a position difference between a second tracking frame corresponding to the third tracking frame and a second tracking frame corresponding to the first tracking frame is not less than a position difference threshold, the camera is updated from the current position to a current target position corresponding to the current target posture in the world coordinate system.

5. The method according to claim 1, characterized in that After obtaining the second tracking frame corresponding to the first tracking frame in the current image, the method further includes: Determining a current position of the first tracking frame in the world coordinate system; Determining a current position of the first tracking frame in a camera coordinate system based on a second tracking frame corresponding to the first tracking frame and the first tracking frame; Based on the current pose of the first tracking frame in the world coordinate system and the current pose of the first tracking frame in the camera coordinate system, a current target pose of the camera in the world coordinate system is determined.

6. The method according to claim 5, characterized in that The determining, based on a second tracking frame corresponding to the first tracking frame and the first tracking frame, a current position and posture of the first tracking frame in a camera coordinate system includes: determining a current depth of the first tracking frame in the camera coordinate system based on a side length of the first tracking frame, a current physical focal length of the camera, and a size of a second tracking frame corresponding to the first tracking frame; Determine a current position of the first tracking frame in the camera coordinate system based on a current depth of the first tracking frame in the camera coordinate system, an internal parameter of the camera, and a current position of a second tracking frame corresponding to the first tracking frame in the world coordinate system; The current position of the first tracking frame in the camera coordinate system and the current posture of the first tracking frame in the camera coordinate system are combined to form the current posture of the first tracking frame in the camera coordinate system.

7. The method according to claim 5, characterized in that The determining the current position of the first tracking frame in the world coordinate system includes: Determining an initial position and posture of the first tracking frame in the camera coordinate system; Determining an initial position of the camera in the world coordinate system; Based on the initial pose of the camera in the world coordinate system and the initial pose of the first tracking frame in the camera coordinate system, a current pose of the first tracking frame in the world coordinate system is determined.

8. The method according to claim 7, characterized in that Determining the initial position of the camera in the world coordinate system includes: Determining an initial position of the camera in the world coordinate system; Acquire initial gyroscope data of the camera corresponding to the first tracking frame, and determine an initial posture of the camera corresponding to the initial gyroscope data; The initial position of the camera in the world coordinate system and the initial posture of the camera are combined into an initial posture of the camera in the world coordinate system.

9. The method according to claim 7, characterized in that: The determining an initial position of the first tracking frame in the camera coordinate system includes: Determine an initial depth of the first tracking frame in the camera coordinate system based on a horizontal plane normal vector, an internal parameter of the camera, and an initial position of the first tracking frame in the world coordinate system; An initial position of the first tracking frame in the camera coordinate system is determined based on an initial depth of the first tracking frame in the camera coordinate system, internal parameters of the camera, and an initial position of the first tracking frame in the world coordinate system.

10. The method according to claim 1, characterized in that The method further comprises: When the movement speed of the camera is greater than the first speed threshold, determining a current posture of the camera in the world coordinate system based on current gyroscope data of the camera; The current posture of the camera in the world coordinate system and the current position of the camera in the world coordinate system are combined to form the current target posture of the camera in the world coordinate system.

11. The method according to claim 1, characterized in that: The determining, based on the second tracking frame in the current image corresponding to the third tracking frame, the current target pose of the camera in the world coordinate system comprises: Determining a current position of the third tracking frame in the world coordinate system; Determine a current position of the third tracking frame in a camera coordinate system based on a second tracking frame corresponding to the third tracking frame in the current image and the third tracking frame; Based on the current pose of the third tracking frame in the world coordinate system and the current pose of the third tracking frame in the camera coordinate system, a current target pose of the camera in the world coordinate system is determined.

12. The method according to claim 11, characterized in that The determining, based on the second tracking frame corresponding to the third tracking frame in the current image and the third tracking frame, a current position and posture of the third tracking frame in the camera coordinate system comprises: Determine a current depth of the third tracking frame in the camera coordinate system based on a side length of the third tracking frame, a current physical focal length of the camera, and a size of a second tracking frame corresponding to the third tracking frame; Determine a current position of the third tracking frame in the camera coordinate system based on a current depth of the third tracking frame in the camera coordinate system, an internal parameter of the camera, and a current position of a second tracking frame corresponding to the third tracking frame in the world coordinate system; The current position of the third tracking frame in the camera coordinate system and the current posture of the camera in the camera coordinate system are combined to form the current posture of the third tracking frame in the camera coordinate system.

13. The method according to claim 1, characterized in that After obtaining the second tracking frame corresponding to the first tracking frame in the current image, the method further includes: When a second tracking frame corresponding to the first tracking frame is located in a boundary area of ​​the current image, acquiring a fourth tracking frame of the current image; The fourth tracking frame is adjacent to the second tracking frame and is closer to the center position of the current image than the second tracking frame; The second tracking frame is updated based on the fourth tracking frame.

14. A camera posture processing device, characterized in that: The device comprises: An initial module, configured to obtain an initial image including a first tracking frame, wherein the initial image is acquired when the camera is first positioned to the first tracking frame; A training module, configured to train a target detector for identifying the initial content based on the initial content of the first tracking frame of the initial image; A current module is used to obtain a current image, wherein the current image is acquired by the camera in real time; obtain current gyroscope data when the camera acquires the current image, and determine the movement speed of the camera based on the current gyroscope data; A repositioning module is used to perform positioning processing on the current image based on the initial content through the target detector when the movement speed of the camera is not greater than a first speed threshold and the training process of the target detector is completed, so as to obtain a second tracking frame corresponding to the first tracking frame in the current image; when the overlapping area between the second tracking frame of the first tracking frame and the first tracking frame is less than the overlapping area threshold, perform tracking processing on the current image based on a third tracking frame corresponding to the first tracking frame in a previous frame of the current image, so as to obtain a second tracking frame corresponding to the third tracking frame in the current image; obtain a tracking score corresponding to the tracking processing, and obtain a tracking score threshold that is negatively correlated with the movement speed of the camera. A determination module is used to determine, when the tracking score exceeds the tracking score threshold, the current target pose of the camera in the world coordinate system based on the second tracking frame corresponding to the third tracking frame in the current image, wherein the current target pose is a camera pose with six degrees of freedom; when the tracking score does not exceed the tracking score threshold, determine the current pose of the camera in the world coordinate system based on the current gyroscope data of the camera, wherein the current pose is a camera pose with three degrees of freedom; and form the current pose of the camera in the world coordinate system and the current position of the camera in the world coordinate system into the current target pose of the camera in the world coordinate system.

15. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, configured to implement the camera posture processing method according to any one of claims 1 to 13 when executing the executable instructions stored in the memory.

16. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by the processor, the camera posture processing method described in any one of claims 1 to 13 is implemented.

17. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the camera posture processing method according to any one of claims 1 to 13 is implemented.

Images