Embodied intelligence architecture-based automatic follow-up control method and system, and medium

Abstract

The application discloses an automatic follow-up shooting control method and system based on an embodied intelligent architecture, and a medium, relates to the technical field of automatic follow-up shooting, and the method comprises the following steps: target tracking is performed on an RGB image, the real position, the real area, the human body posture category, the human body face direction and the human body face expression of a human body to be shot are determined, an obstacle occupation grid map is constructed based on a depth image, text conversion is performed on a voice input instruction to obtain input text, a large language model is further utilized to determine the expected position and the expected area of the human body to be shot and output text, and motion control instructions and voice output instructions are further generated, the motion control instructions are used for controlling the motion of a shooting device, the voice output instructions are used for shooting guidance on the human body to be shot, and shooting is completed. The application can deeply integrate "environment perception", "aesthetic judgment" and "camera movement decision", and improve the shooting effect.

Description

Technical Field

[0001] This application relates to the field of automatic tracking technology, and in particular to an automatic tracking control method, system and medium based on an embodied intelligence architecture. Background Technology

[0002] With the widespread adoption of smart photography devices, automatic tracking technology has been widely used in live streaming, video conferencing, and personal video recording. Current automatic tracking control methods mainly rely on visual servoing technology, whose core logic is to lock onto the target using computer vision algorithms and control camera movement to reduce the target's positional error in the frame.

[0003] However, the limitations of mainstream automatic tracking control methods are as follows: (1) Center-locking algorithm based on detection box This is the most commonly used algorithm in consumer-grade smart gimbals (such as DJI and Hohem products). Its logic is to use an object detection network (such as MobileNet-SSD) to obtain the bounding box of the target, calculate the deviation between its center point and the center of the image, and use a PID controller to drive the motor to eliminate the deviation, so that the target is always in the center of the image. However, its limitations are: 1) Rigid composition: The algorithm only pursues "centering" and does not understand photographic aesthetics at all. For example, when shooting a side profile of a person, it cannot leave space for the line of sight according to the direction of the line of sight; when shooting a full-body photo, it cannot adjust the position of the person according to the golden ratio; 2) Lack of environmental semantic understanding: The algorithm cannot perceive the clutter of the background. Even if there is a trash can or strong light interference in the background, the algorithm will still mechanically lock the person in front of the current background and cannot optimize the background by moving the camera position like a human photographer.

[0004] (2) Visual SLAM and path planning algorithm In the field of robotics, visual SLAM and path planning algorithms (VSLAM, such as ORB-SLAM) are mainly used for localization and navigation to ensure that the robot moves from point A to point B without collision. However, their limitations are: the task objective is singular, generally focusing on "shortest path" or "obstacle avoidance" as the optimization objective, rather than "best shooting angle" as the optimization objective. They have no understanding of photographic aesthetics, do not know which position has better lighting, or which angle can capture a more visually impactful image.

[0005] (3) Rule-based graph construction-aided algorithm Some cameras have "composition guidelines" functionality, but these only provide composition functions and do not have the ability to intelligently adjust shooting parameters. Their limitation is that they are passive; these functions are usually passive auxiliary displays and cannot actively generate control commands to drive the camera to change its position, angle, or other shooting parameters to capture the best moment.

[0006] Based on this, the relevant automatic tracking control methods have the problem of "body-brain separation". The visual algorithm is only responsible for "seeing people clearly", and the control algorithm is only responsible for "not hitting the wall" or "keeping up with people". There is a lack of an embodied intelligent architecture that can deeply integrate "environmental perception", "aesthetic judgment" and "camera movement decision" like a human photographer. Summary of the Invention

[0007] The purpose of this application is to provide an automatic tracking control method, system, and medium based on an embodied intelligence architecture, which can deeply integrate "environmental perception", "aesthetic judgment" and "camera movement decision" to improve shooting results.

[0008] To achieve the above objectives, this application provides the following solution.

[0009] In a first aspect, this application provides an automatic tracking control method based on an embodied intelligence architecture, the automatic tracking control method based on an embodied intelligence architecture comprising: Before photographing the human body, acquire RGB images, depth images, and voice input commands; the RGB images and depth images are images obtained by photographing the environment in which the human body is located, and the voice input commands are voices issued by the human body to express the shooting needs; Target tracking is performed on the RGB image to determine the detection box corresponding to the human body to be photographed. Based on the detection box corresponding to the human body to be photographed, the true position, true area, human pose category, human face orientation, and human facial expression of the human body to be photographed are determined. Construct an obstacle occupancy grid map based on depth images; The voice input command is converted into text to obtain the input text; Using real location, real area, human pose category, human face orientation, human face expression, obstacle occupancy grid map and input text as input, a large language model is used to determine the expected position and expected area of ​​the human body to be photographed and the output text; the large language model is used to make camera movement decisions based on photographic aesthetics. Based on the actual position, actual area, desired position, and desired area, motion control commands are generated. The output text is then converted into speech to obtain speech output commands. The motion control commands are used to control the movement of the shooting device so that the human body to be shot is in the desired position of the shooting frame and occupies the desired area of ​​the shooting frame. The speech output commands are used to provide shooting guidance for the human body to be shot and complete the shooting.

[0010] Secondly, this application provides an automatic follow-shooting control system based on an embodied intelligence architecture. The automatic follow-shooting control system based on an embodied intelligence architecture includes: an image acquisition unit, a voice acquisition unit, a control unit, and a voice playback unit. The image acquisition unit, the voice acquisition unit, and the voice playback unit are all installed on the shooting device, and the image acquisition unit, the voice acquisition unit, and the voice playback unit are all communicatively connected to the control unit. The image acquisition unit is used to acquire RGB images and depth images; The voice acquisition unit is used to acquire voice input commands; The control unit is used to execute the above-mentioned automatic tracking control method based on embodied intelligence architecture, and obtain motion control commands and voice output commands. The motion control commands are used to control the movement of the shooting device so that the human body to be shot is in the desired position of the shooting frame and occupies the desired area of ​​the shooting frame. The voice output commands are used to provide shooting guidance for the human body to be shot. The voice playback unit is used to play voice output commands.

[0011] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described automatic tracking control method based on embodied intelligence architecture.

[0012] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides an automatic tracking control method, system, and medium based on an embodied intelligence architecture. Before filming the human subject, RGB images, depth images, and voice input commands are acquired. The RGB and depth images are images obtained by filming the environment in which the human subject is located. The voice input command is the voice expression of the human subject expressing the filming needs. Target tracking is performed on the RGB images to determine the detection box corresponding to the human subject. Based on the detection box, the true position, true area, human pose category, human face orientation, and human facial expression of the human subject are determined. An obstacle occupancy grid map is constructed based on the depth image. Finally, the voice input command is converted into text. The system receives input text, including the actual location, actual area, human pose category, human face orientation, human facial expression, obstacle occupancy grid map, and the input text itself. A large language model is used to determine the desired location and area of ​​the human subject to be photographed, as well as the output text. The large language model is used for camera movement decisions based on photographic aesthetics. Based on the actual location, actual area, desired location, and desired area, motion control commands are generated. The output text is then converted into speech to obtain speech output commands. These motion control commands control the movement of the shooting equipment, ensuring the human subject is positioned in the desired location within the frame and occupies the desired area. The speech output commands provide shooting guidance for the human subject, completing the shooting process. This application acquires RGB images, depth images, and voice input commands, enabling environmental awareness. It introduces a large language model, which is used to make camera movement decisions based on photographic aesthetics and user intent (represented through input text). At this point, based on the real location, real area, human posture category, human face orientation, human facial expression, obstacle occupancy grid map (able to understand the background), and input text, it determines the desired position and area of ​​the human being to be photographed, as well as the output text. It can make aesthetic judgments and camera movement decisions, actively adjust the shooting position, and actively provide shooting guidance, solving problems such as rigid composition, lack of environmental semantic understanding, single task objective, passivity, and lack of interaction (due to the ability to provide shooting guidance). It can deeply integrate "environmental awareness," "aesthetic judgment," and "camera movement decisions" to improve shooting effects. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 An application environment diagram of an automatic tracking control method based on an embodied intelligence architecture provided in an embodiment of this application; Figure 2 A flowchart illustrating an automatic tracking control method based on an embodied intelligence architecture, provided as an embodiment of this application; Figure 3 A schematic diagram of the technical route of an automatic tracking control method based on an embodied intelligence architecture provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] The automatic tracking control method based on embodied intelligence architecture provided in this application embodiment can be applied to, for example... Figure 1The application environment shown depicts a scenario where the terminal communicates with the server via a network. The data storage system stores the data the server needs to process. This data storage system can be configured independently, integrated into the server, or located in the cloud or on another server. The terminal can send control requests to be processed to the server. Upon receiving the control request, the server, before capturing the subject, acquires an RGB image, a depth image, and a voice input command. It performs target tracking on the RGB image to determine the detection box corresponding to the subject. Based on the detection box, it determines the subject's true position, true area, pose type, facial orientation, and facial expression. Based on the depth image, it constructs an obstacle occupancy grid map. It performs text conversion on the voice input command to obtain the input text. Using the true position, true area, pose type, facial orientation, facial expression, obstacle occupancy grid map, and input text as input, it uses a large language model to determine the desired position and desired area of ​​the subject, as well as the output text. Based on the true position, true area, desired position, and desired area, it generates motion control commands. It then performs speech conversion on the output text to obtain voice output commands. These motion control commands control the movement of the capturing device, ensuring the subject is positioned in the desired location and occupies the desired area within the captured frame. The voice output commands provide guidance for capturing the subject. The server can feed back the control results, namely the motion control commands and voice output commands received in response to the control request, to the terminal.

[0018] Furthermore, in some embodiments, the automatic tracking control method based on the embodied intelligence architecture can also be implemented by a server or a terminal independently. For example, the terminal can directly process the control request to be processed, or the server can obtain the control request to be processed from the data storage system and process it.

[0019] The terminals can be, but are not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Servers can be implemented using independent servers, server clusters composed of multiple servers, or cloud servers.

[0020] In one exemplary embodiment, such as Figure 2 As shown, an automatic tracking control method based on an embodied intelligence architecture is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1The following steps, S1 to S6, are used as an example to illustrate the process of using a server in the example.

[0021] Step S1: Before shooting the human body, acquire RGB image, depth image and voice input command; RGB image and depth image are images obtained by shooting the environment in which the human body is located, and voice input command is the voice expression of the human body expressing the shooting needs.

[0022] Step S2: Perform target tracking on the RGB image to determine the detection box corresponding to the human body to be photographed. Based on the detection box corresponding to the human body to be photographed, determine the true position, true area, human pose category, human face orientation, and human facial expression of the human body to be photographed.

[0023] Step S3: Based on the depth image, construct an obstacle-occupied grid map.

[0024] Step S4: Convert the voice input command into text to obtain the input text.

[0025] Step S5: Using the real location, real area, human pose category, human face orientation, human face expression, obstacle occupancy grid map and input text as input, the large language model is used to determine the expected position and expected area of ​​the human body to be photographed and the output text; the large language model is used to make camera movement decisions based on photographic aesthetics.

[0026] Step S6: Based on the actual position, actual area, desired position, and desired area, generate motion control instructions, convert the output text into speech, and obtain speech output instructions. The motion control instructions are used to control the movement of the shooting device so that the human body to be shot is in the desired position of the shooting frame and occupies the desired area of ​​the shooting frame. The speech output instructions are used to provide shooting guidance for the human body to be shot and complete the shooting.

[0027] By implementing steps S1 to S6 above, this embodiment can deeply integrate "environmental perception", "aesthetic judgment" and "camera movement decision" to improve the shooting effect.

[0028] The following, combined with Figure 3 This paper provides a detailed description of an automatic tracking control method based on an embodied intelligence architecture used in this embodiment.

[0029] (a) Data collection In this embodiment, before photographing the human body, RGB images, depth images, and voice input commands are acquired. The RGB images and depth images are images obtained by photographing the environment in which the human body is located, and the voice input commands are voices issued by the human body expressing the photographing needs.

[0030] This embodiment designs a visual acquisition unit, which is used to acquire RGB images and depth images. For example, the visual acquisition unit can be a depth camera or a binocular camera that can simultaneously acquire image information and depth information, thereby acquiring visual stream data of the shooting scene.

[0031] This embodiment designs a voice interaction unit, which is used to collect voice input commands issued by the user (i.e., the human body to be photographed), and can subsequently play the generated voice output commands. For example, the voice interaction unit may include a microphone array and a speaker. The microphone array is used to collect voice input commands generated by the user expressing shooting needs in natural language in real time, and the speaker is used to play the generated voice output commands, which can provide voice guidance on the user's posture and emotions.

[0032] Through the aforementioned visual acquisition unit and voice interaction unit, multimodal acquisition and interaction can be completed, real-time acquisition of environmental audio data, high-resolution video streams and depth information data can be acquired, and multimodal perception can be achieved, that is, environmental perception can be realized.

[0033] This embodiment can use a control unit to execute the automatic tracking control method. The control unit can adopt a hardware and software collaborative architecture of "edge computing platform (host computer) + microcontroller (lower computer)". That is, the control unit can be divided into a host computer (edge ​​computing and intelligent decision-making core) and a lower computer (lower-level motion control core). The host computer is the intelligent decision-making center and can use an edge computing platform equipped with a high-performance GPU (such as NVIDIA Jetson Orin or a discrete graphics card computing unit) to be responsible for visual perception, 3D mapping, large language model inference and motion smoothing calculation. The lower computer can use a microcontroller (MCU), specifically an STM32F4 / F7 series ARM Cortex-M microcontroller, to be responsible for low-level motion driving.

[0034] The aforementioned visual acquisition unit can employ an RGB-D depth camera, integrated at the top of the imaging device. It can connect to a host computer via a USB 3.0 interface for communication, maintaining a frame rate of ≥30fps and simultaneously outputting RGB color image streams and depth image streams to obtain RGB and depth images. The aforementioned voice interaction unit can employ a directional microphone array and speaker, integrated at the top of the imaging device. It can connect to a host computer via an I2S or USB audio interface for communication, acquiring voice input commands and playing voice output commands, while also supporting sound source localization and environmental noise reduction.

[0035] This embodiment targets an existing shooting device, which has an omnidirectional actuator and a camera. The camera is used to capture images of the human body after the shooting device executes motion control commands and voice output commands. The camera can reuse the visual acquisition unit, that is, the visual acquisition unit can be directly used as the camera. The omnidirectional actuator is implemented through a chassis topology, using a four-wheel independently driven omnidirectional mobile chassis (hereinafter referred to as the chassis). The chassis includes four drive wheels, each of which can be a Mecanum wheel. Each drive wheel is equipped with a brushless DC motor with a Hall encoder. The brushless DC motor is used to drive the drive wheel. The motor driver of each brushless DC motor is connected to the lower-level machine and communicates with the lower-level machine. The lower-level machine controls the underlying motor driver through CAN bus or PWM direct drive, thereby driving the drive wheel.

[0036] The shooting device in this embodiment may also include a multi-axis gimbal, which is mounted on a chassis and can drive the camera to rotate.

[0037] It should be noted that "before filming the person to be filmed" refers to the moment when it is determined that the person to be filmed needs to be filmed. For example, when a filming instruction is received from the person to be filmed, it can be determined that the person to be filmed needs to be filmed. This is the initiation time "before filming the person to be filmed", and the subsequent process begins. The filming instruction issued by the person to be filmed can be a voice instruction that indicates the start of filming, such as "start filming" or "filming while walking". It can be obtained through the voice interaction unit and recognized by the control unit. The specific process is a mature existing technology and will not be elaborated here.

[0038] (II) RGB Image Processing In this embodiment, target tracking is performed on the RGB image to determine the detection box corresponding to the human body to be photographed. Based on the detection box corresponding to the human body to be photographed, the true position, true area, human posture category, human face orientation, and human facial expression of the human body to be photographed are determined.

[0039] This embodiment performs target anchoring and state and emotion analysis (i.e., visual perception) based on RGB images. It identifies and locks the target subject (i.e., the human body to be filmed) in the real-time video stream, performs continuous multi-target tracking and trajectory prediction, performs target anchoring, determines the detection box corresponding to the human body to be filmed, and extracts features such as the target subject's skeletal posture, facial orientation, and facial expression. It then performs state and emotion analysis to determine the true position, true area, human posture category, human facial orientation, and human facial expression of the human body to be filmed.

[0040] The aforementioned target anchoring is accomplished by the visual tracking module of the host computer. This visual tracking module receives the RGB image acquired by the visual acquisition unit, uses target detection and multi-target tracking algorithms to lock the unique identity features of the target subject, determines the detection box corresponding to the human body to be photographed, achieves highly robust visual tracking, and solves the problem of "where did the target subject go?"

[0041] When performing target tracking, this embodiment can employ a multi-target tracking algorithm based on the fusion of appearance and motion features, such as BoT-SORT, as follows: (1) Generation of detection box set: Input the RGB image into a lightweight detection network (such as YOLOv8), and output the detection box corresponding to each candidate target in the k-th frame of the RGB image to form a detection box set. , among which, the Detection boxes corresponding to each candidate target , The first The x-coordinate, y-coordinate, width, and height of the center point of the detection box corresponding to each candidate target.

[0042] (2) Feature vector set generation: Input the detection box corresponding to each candidate target in the k-th frame RGB image into the appearance feature extraction model (an existing appearance feature extraction network, such as the Re-ID network, can be used), and output the appearance feature vector corresponding to each candidate target in the k-th frame RGB image. (If a Re-ID network is used, the appearance feature vector is the Re-ID appearance feature vector output by the Re-ID network), forming a feature vector set.

[0043] (3) Kinematic prediction: Based on the historical trajectory set (including the historical trajectory corresponding to each candidate target in the (k-1)th frame RGB image), the Kalman filter is used to predict the state vector of the target in the kth frame RGB image. This is the prior state vector, which includes the predicted detection box corresponding to the human body to be photographed. The state matrix, This is the historical state vector of the target subject in the (k-1)th frame of the RGB image.

[0044] (4) Feature matching and anti-occlusion: For each candidate target's corresponding detection box in the detection box set, calculate the Mahalanobis distance between the candidate target's corresponding detection box and the predicted detection box of the human body to be photographed. Matching is performed based on the Mahalanobis distance. If the Mahalanobis distance is less than a preset threshold, the candidate target's corresponding detection box is retained as a candidate detection box, and the Mahalanobis distance of each candidate detection box is recorded. For each candidate detection box, the candidate detection box and the predicted detection box are paired, and matching is performed based on the intersection over union (IoU). The intersection-union ratio (IUGR) for each candidate detection box is calculated as follows: IUGR = candidate detection box area + predicted detection box area - overlap area between candidate and predicted detection boxes. For each candidate detection box, matching is performed based on the appearance feature vector, and the cosine similarity between the candidate detection box and the historical trajectory is calculated to obtain the cosine similarity for each candidate detection box.

[0045] The formula for calculating cosine similarity is: ; in, Cosine similarity; The appearance feature vector of the detection box corresponding to the target subject in the most recent RGB image from the historical trajectory of the target subject (the historical trajectory is the trajectory formed by the detection boxes of the target subject in each frame of RGB images); This represents the appearance feature vector corresponding to the candidate detection box.

[0046] Based on the Mahalanobis distance, intersection-union ratio, and cosine similarity corresponding to each candidate detection box, bipartite graph matching is performed using the Hungarian Algorithm to determine the detection box corresponding to the target subject in the k-th frame RGB image.

[0047] In this embodiment, target tracking is performed on the RGB image to determine the detection box corresponding to the human body to be photographed, specifically including: (1) Using the RGB image as input, the target detection model is used to determine the detection box set, which includes the detection box corresponding to each candidate target in the RGB image.

[0048] The target detection model can use a lightweight detection network.

[0049] (2) Take the detection box corresponding to each candidate target in the RGB image as input, use the appearance feature extraction model to determine the appearance feature vector corresponding to each candidate target in the RGB image, and form a feature vector set by combining the appearance feature vectors corresponding to each candidate target in the RGB image.

[0050] (3) Based on the historical trajectory set, the state vector corresponding to the human body to be photographed in the RGB image is determined by using a Kalman filter. The state vector includes the predicted detection box corresponding to the human body to be photographed.

[0051] (4) Based on the set of detection boxes, the set of feature vectors, the set of historical trajectories and the state vector, the Hungarian algorithm is used to perform bipartite graph matching to obtain the detection box corresponding to the human body to be photographed.

[0052] The aforementioned state and emotion analysis is performed by the visual analysis module of the host computer. This visual analysis module uses human pose estimation models (such as OpenPose, MediaPipe Pose, or YOLOv8-pose) to calculate the coordinates of skeletal key points, determine the human pose category and human face orientation, and use emotion classification models or facial expression recognition models to determine human facial expressions.

[0053] In this embodiment, based on the detection box corresponding to the human body to be photographed, the true position, true area, human posture category, human face orientation, and human facial expression of the human body to be photographed are determined, specifically including: (1) Based on the detection frame corresponding to the human body to be photographed, determine the real position and real area of ​​the human body to be photographed.

[0054] The true position can be the coordinates of the center point of the detection frame corresponding to the human body to be photographed, and the true area can be the area of ​​the detection frame corresponding to the human body to be photographed.

[0055] (2) Using the detection box corresponding to the human body to be photographed as input, the coordinates of the human body key points are determined by the human body pose estimation model, and the human body pose category and human face orientation are determined based on the coordinates of the human body key points.

[0056] The human pose estimation model uses an improved YOLOv8-pose, which is obtained by adding a CBAM attention mechanism module after each C2f module in the backbone network of YOLOv8-pose.

[0057] Key points of the human body can be defined according to the COCO standard, including the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0058] The process of determining the human posture category and facial orientation of the subject based on the coordinates of key points on the human body is a mature existing technology and will not be elaborated here. The human posture category can be standing, running, squatting, sitting, etc., and the facial orientation can be horizontal (left, front, or right).

[0059] (3) Using the detection box corresponding to the human body to be photographed as input, the facial expression of the human body to be photographed is determined by the emotion classification model or the facial expression recognition model.

[0060] The emotion classification model and facial expression recognition model both use existing models, which will not be elaborated on here.

[0061] The actual position, actual area, posture type, facial orientation, and facial expression of the human body to be photographed constitute the state of the target subject.

[0062] (III) Depth Image Processing In this embodiment, an obstacle-occupying grid map is constructed based on depth images.

[0063] This embodiment uses depth information and visual odometry (i.e., using continuous RGB images acquired by the visual acquisition unit to estimate the displacement and rotation of the camera and output the inter-frame pose transformation matrix) to perform 3D mapping, and builds a 3D octree map (i.e., obstacle-occupied grid map) containing depth information in real time to realize 3D scene reconstruction and complete the three-dimensional spatial perception of the shooting scene.

[0064] The aforementioned 3D mapping is completed by the environment mapping module of the host computer. Based on the RGB images and depth images acquired by the visual acquisition unit, the environment mapping module constructs a 3D octree map containing depth information. This 3D octree map is not only used to record the three-dimensional spatial position of the target subject, but also provides a spatial data foundation for environmental cognition, scene segmentation and subsequent obstacle avoidance motion smoothing calculation.

[0065] The 3D mapping process can be implemented using SLAM algorithms, specifically Octomap, ORB-SLAM3, Cartographer, or RTAB-Map, to solve the problem of "what does the environment look like" and achieve 3D spatial perception.

[0066] The process of constructing a 3D octree map containing depth information is as follows: (1) Extract the depth image from the depth camera, convert the depth image into point cloud data, and combine it with the inter-frame pose transformation matrix obtained by visual odometry. The local point cloud data is converted to the world coordinate system to obtain the global point cloud data.

[0067] (2) Based on the global point cloud data, the continuous three-dimensional space is divided into a three-dimensional octree mesh, which includes multiple voxel nodes. Each voxel node is updated based on the logarithmic probability update formula of the occupancy probability. The log-odds of occupancy probability: ; in, voxel nodes The logarithm of the probability of occupancy. Including the depth images from frame 1 to frame t, where the depth image of frame t is the depth image of the current frame; voxel nodes The logarithmic odds of historical occupancy probability. Including depth images from frame 1 to frame t-1; voxel nodes The log-odds increment of the probability of occupancy. This is the depth image of frame t, i.e., the sensor observation value of the current frame; This is the logarithmic probability of the initial prior occupancy probability.

[0068] For each voxel node Based on voxel nodes The log odds of occupancy probability determine voxel nodes. The state is occupied by an obstacle, free, or unknown, if the voxel node If the log-odds ratio of the occupancy probability is greater than the first log-odds threshold, the state is occupied by an obstacle. If the voxel node... If the log-odds ratio of the occupancy probability is less than the second log-odds ratio threshold, the state is idle; otherwise, the state is unknown.

[0069] Based on this, it can be clearly identified which spaces are drivable (Free) areas and which are obstacles (Occupied), providing physical constraints for subsequent safe obstacle avoidance.

[0070] The aforementioned obstacles occupy the grid map, forming the environmental state.

[0071] (iv) Voice input command processing In this embodiment, the voice input command is converted into text to obtain the input text.

[0072] This embodiment can use any existing speech-to-text model for text conversion, such as the Whisper model.

[0073] The above input text constitutes the user's shooting intention.

[0074] (v) Large Language Model Reasoning In this embodiment, the actual location, actual area, human posture category, human face orientation, human face expression, obstacle occupancy grid map and input text are used as inputs. The large language model is used to determine the expected location and expected area of ​​the human body to be photographed and the output text. The large language model is used to make camera movement decisions based on photographic aesthetics.

[0075] This embodiment uses a large language model for multimodal aesthetic evaluation and intent parsing. It completes large language model reasoning by inputting the target subject's position (actual position), area (actual area), posture (human posture category and human face orientation), and emotion (human facial expression), 3D scene structure information (obstacles occupying the grid map), and the user's natural language interaction commands (input text) into a large language model with a photographic aesthetic rule base. Through deep fusion of semantics and scene, it dynamically calculates and outputs the current optimal aesthetic composition reference point (which may not be the geometric center) and the expected shooting distance, determines the expected position and expected area of ​​the human body to be photographed, and outputs the text.

[0076] The aforementioned large language model inference is completed by the multimodal aesthetic evaluation and decision-making module in the host computer. This multimodal aesthetic evaluation and decision-making module has a built-in large language model (LLM) and a photographic aesthetics rule library. It integrates the real-time position, area, posture and emotional features of the target subject, the environmental spatial structure fed back by the 3D octree map (i.e. 3D scene structure information), and the user's natural language interaction commands to perform multimodal joint inference, actively conduct aesthetic evaluation, and dynamically determine shooting strategy control commands (such as: switching the rule of thirds composition, changing the framing ratio, foreground white space, etc.). It completely abandons the traditional forced centering image locking logic, determines the expected position and expected area of ​​the human body to be photographed, and outputs text.

[0077] Traditional automatic tracking control methods use hard-coded locking of the image center. This embodiment introduces a large language model to realize an "intelligent agent" with photographic aesthetic knowledge. The process of multimodal aesthetic evaluation and decision-making driven by the large language model is as follows: (1) Structured scene description and Prompt (prompt word) engineering (How does a large language model know which parts are good or bad?) Since the large language model itself cannot directly "see" and understand the video, this embodiment uses a preprocessor algorithm to structure (JSONify) the scene and inject it into the Prompt context of the large language model.

[0078] The context state of the input large language model includes: 1) System status: {"target_locked (target lock flag)": true (including the actual position and area of ​​the target), "current_composition (current screen composition mode) (optional)": "center"}. The values ​​of current_composition (current screen composition mode) can also be: "third-line", "foreground-margin", "close-up", "full-scene".

[0079] 2) Visual semantic extraction: Use human pose estimation models to determine the orientation of human faces and the category of human poses, for example, to determine that "the user is looking to the right of the screen and is walking to the right". Use emotion classification models to determine human facial expressions, for example, to determine that "the user's expression is stiff".

[0080] 3) Obstacles occupy the grid map.

[0081] 4) Voice command parsing: The Whisper model is used to transcribe the user's voice input commands into the input text, such as "The screen is too crowded, move back a little to take a picture".

[0082] 5) User's desired style (optional): A style description or reference provided by the user, such as "I want photos in the style of Hideaki Hamada" or "I want photos that are dynamic and vivid".

[0083] The above options can be entered by the user, or the default value can be used directly if no user input is received.

[0084] At this point, the constructed large language model Prompt input example is: "You are a professional robotic photographer. The current subject is located in the center of the frame, occupying 60% of the frame. Environment description: There is a vast landscape on the right (provided by 3D octree map semantics), the subject is looking to the right, and has a stiff expression. The user issues a verbal instruction: 'The frame is too crowded, move back a bit to take the shot.' Please combine the rule of thirds and the aesthetics of negative space to output the next shooting control parameters."

[0085] (2) Aesthetic reasoning and JSON action code output of large language models: The large language model makes logical inferences based on the photography aesthetics rule library internalized during pre-training: since the user is looking to the right, "guiding space" should be left on the right side, so the target subject should not be in the center, but should be moved to the left third line; since the user feels "crowded", the physical distance needs to be increased (the area ratio needs to be reduced); since the user's expression is stiff, voice interaction is needed.

[0086] The final structured JSON action code output by the large language model is as follows: { "reasoning (compositional anchor point)": "If the user's gaze is to the right and they want to look back, switch to a left-right rule-of-flight composition to allow for visual space and reduce the area occupied to create distance. For stiff facial expressions, trigger a soothing voice message." "composition_target (the center position of the target subject in the frame)": { "x_ratio (the percentage of the desired x-coordinate)": 0.333, / / The desired center of the subject is located at 33.3% of the screen width (the left third line). "y_ratio (the percentage of the desired y-coordinate)": 0.5, / / The desired center of the subject is at 50% of the screen height. }, "scale_target (shot zoom target)": { "area_ratio (expected percentage)": 0.4, / / Expected target body boundary area percentage to decrease from 60% to 40% (controlling chassis retreat) }, "tts_feedback (voice feedback output, i.e., output text)": "Okay, I've stepped back and adjusted the composition for you. Relax and smile!" }

[0087] The x-coordinate of the desired location can be determined by the proportion of the x-coordinate of the desired location, the y-coordinate of the desired location can be determined by the proportion of the desired location, and the desired area can be determined by the desired proportion.

[0088] (vi) Control output In this embodiment, motion control commands are generated based on the actual position, actual area, desired position, and desired area. The output text is converted into speech to obtain speech output commands. The motion control commands are used to control the movement of the shooting device so that the human body to be shot is in the desired position of the shooting frame and occupies the desired area of ​​the shooting frame. The speech output commands are used to provide shooting guidance for the human body to be shot and complete the shooting.

[0089] This embodiment first performs smooth motion planning under spatial constraints, calculates the deviation between the actual position and actual area of ​​the target subject and the desired aesthetic benchmark (i.e., desired position and desired area), combines the obstacle avoidance constraints provided by the 3D octree map and the set dead zone anti-shaking threshold, performs trajectory smoothing processing, and outputs a safe physical displacement velocity vector.

[0090] The above process is motion smoothing calculation, which is completed by the motion planning module with dead zone stabilization in the host computer. This motion planning module receives the expected aesthetic reference point output by the multimodal aesthetic evaluation and decision module, and calculates multi-dimensional visual or spatial deviations by combining the actual position and actual area of ​​the target subject. It introduces dead zone control and smoothing filtering algorithms to filter high-frequency visual jitter noise. At the same time, it combines the barrier-free driving space corresponding to the 3D octree map to safely and smoothly map the deviation of the two-dimensional image into the three-dimensional motion velocity vector (i.e., physical displacement velocity vector) in the physical space.

[0091] After obtaining the mapping parameters x_ratio, y_ratio, and area_ratio from the large language model, the motion planning module is responsible for converting them into physical displacement-velocity vectors. The motion planning algorithm with dead-zone stabilization and 3D constraints is as follows: (1) Calculation of multidimensional visual or aesthetic deviation Let the resolution of the current frame's RGB image be... , For width, The height is given by the coordinates of the center point of the bounding box corresponding to the target object. (It is the actual location), with an area of (This is the actual area), and the horizontal mapping error (also known as yaw mapping error) is calculated using the first error calculation formula. The distance error (also known as the scale error) is calculated using the second error calculation formula. : The formula for calculating the first error is: ; in, This refers to horizontal composition error; The x-coordinate of the actual location; Let x be the x-coordinate of the desired position. The width of the RGB image. This represents the percentage of the x-coordinate of the desired location.

[0092] The formula for calculating the second error is: ; in, This refers to distance error; This represents the actual area. For the desired area, The height of the RGB image. This represents the expected percentage.

[0093] The horizontal composition error and the distance error are combined into an error set.

[0094] (2) Dead zone control and smoothing filtering algorithm to implement anti-shake filtering mechanism Because the bounding boxes in computer vision inevitably experience pixel-level jitter between consecutive frames, directly sending the error set to the host computer in the control unit would cause high-frequency movements of the chassis, indirectly leading to high-frequency jitter of the gimbal. To solve this problem, this embodiment first defines a dead-time function. : ; in, This represents the actual error. For error, when For horizontal composition error, then For actual horizontal composition error, when For distance error, then This represents the actual distance error. This is the dead zone tolerance threshold (i.e., dead zone stabilization threshold) corresponding to the error. It can be a manually set dead zone tolerance threshold, such as 5% of the pixel width. It is a symbolic function.

[0095] PID control is only considered necessary when the error in the error set exceeds the dead zone tolerance threshold.

[0096] (3) Incremental PID smoothing control The effective actual errors (i.e., actual horizontal mapping error and actual distance error) after dead zone filtering are input into the incremental PID controller to calculate the desired linear velocity (including the desired longitudinal forward and backward velocity). and expected lateral translation speed ) and desired angular velocity (including desired rotational angular velocity) ), which form the physical displacement velocity vector (i.e. the desired velocity set).

[0097] ; in, The velocity at time k; The velocity at time k-1; This is the coefficient for the proportional term; The actual error at time k; This represents the actual error at time k-1; The coefficient of the integral term; These are the coefficients of the differential term; This represents the actual error at time k-2.

[0098] When the actual error is the actual horizontal mapping error, the speeds are the expected lateral translation speed and the expected rotational angular velocity, respectively; when the actual error is the actual distance error, the speed is the expected longitudinal forward and backward speed.

[0099] Before outputting the final speed, the voxel node status in the target velocity vector direction of the 3D octree map is checked. If there is a collision risk, the speed is forcibly reduced proportionally or an obstacle avoidance trajectory is executed. Specifically, the 3D octree map divides the environment space into several voxel nodes. Each voxel node records a state such as idle, occupied, or unknown. Before outputting the final speed, the target velocity vector of the chassis is synthesized based on three velocities (expected lateral translation speed, expected longitudinal forward and backward speed, and expected rotational angular velocity). The spatial area that the shooting device may pass through in a short time (e.g., 1 second, which can be designed by the user) is determined along the target velocity vector direction, and the voxel nodes in the spatial area are checked for occupied status. If occupied status is found, a collision risk is determined, and the three velocities are reduced proportionally or switched to a preset obstacle avoidance trajectory. The proportion can be set by the user, and the obstacle avoidance trajectory can be generated using existing path planning algorithms, prioritizing the generation of obstacle avoidance trajectories in areas where idle voxel nodes are located on the left or right.

[0100] In this embodiment, the physical displacement velocity vector is sent to the lower-level machine, which drives the chassis to perform multi-degree-of-freedom motion, realizing active tracking and composition correction in physical space, and realizing low-level motion drive. At the same time, based on the output text, guiding voice feedback (i.e., voice output commands, such as "please turn a little to the left" or "smile") is synthesized and played through the speaker, forming a complete visual, auditory and motion multimodal control closed loop.

[0101] The aforementioned embodied driving and two-way voice interaction are completed by the underlying execution and closed-loop control module in the lower-level machine. The microcontroller is used as the core of the lower-level machine. It receives the physical displacement velocity vector sent by the upper-level machine, performs closed-loop control through PID algorithm, and solves the physical displacement velocity vector into the PWM control signal of the motor. This drives the chassis to achieve omnidirectional free movement in the plane and actively cooperates to complete the aesthetic composition adjustment.

[0102] The underlying communication protocol and chassis reverse engineering driver are as follows: (1) High-reliability USART custom communication frame structure The host computer will set the desired speed for the planned floating-point type. The data is converted into low-level integer instructions and sent to the lower-level machine via the USART serial bus. The length of a complete data packet is 14 bytes, and the format is defined as follows: [0]: Frame header 1 (0x55); [1]: Frame header 2 (0xAA); [2]: Control command type code (0x01 motion control); [3]-

[11] : Data payload (each occupying 3 bytes of storage) (sign and absolute value);

[12] : Checksum (the lower 8 bits of the sum of all bytes from [2] to

[11] );

[13] : End of frame (0x0D carriage return).

[0103] (2) Realization of inverse kinematics of Mecanum wheel chassis After the lower-level machine receives the data packet and successfully verifies the Checksum, it executes the inverse kinematics solution of the Mecanum wheel.

[0104] Let the longitudinal axis of the shooting device be... The lateral wheel track is The desired speed set at the chassis center is The formulas for calculating the expected wheel speeds (i.e., expected rotational speeds) of the four drive wheels (front left, front right, rear left, and rear right) are as follows: ; ; ; ; in, The desired wheel speed of the first drive wheel (front left); The radius of the drive wheel; The desired longitudinal forward and backward speed; The desired lateral translation speed; The desired angular velocity of rotation; It is half of the longitudinal wheelbase; It is half the lateral track width; The desired wheel speed of the second drive wheel (front right); The desired wheel speed of the third drive wheel (rear left); The desired wheel speed for the fourth drive wheel (right rear).

[0105] The radius of the drive wheel is the same as the radius of the Mecanum wheel.

[0106] The lower-level machine's timer calculates the desired wheel speed. The system dynamically adjusts the duty cycle, generates a PWM signal, drives the brushless DC motor corresponding to the drive wheel, and synthesizes the vector force of the special rollers of the Mecanum wheel. This allows the shooting device to smoothly move left or right while keeping the lens stable and facing the target subject, perfectly executing advanced aesthetic actions such as "left third line composition" issued by the large language model, forming a complete closed loop of "perception-cognition-embodied execution".

[0107] The lower-level machine can also perform two-way voice interaction closed loop: extract the tts_feedback field from the above JSON action code, generate a waveform file through the TTS speech synthesis module, obtain voice output instructions, and play them through the speaker, so as to realize the machine's shooting device to soothe the emotions and guide the posture of the target subject as a human.

[0108] In this embodiment, based on the actual position, actual area, desired position, and desired area, motion control commands are output, specifically including: (1) Based on the true location, true area, expected location and expected area, the error set is calculated.

[0109] (2) Using the error set as input, the desired speed set is determined by the PID controller.

[0110] (3) Based on the expected speed set, the expected wheel speed set is calculated, which includes the expected wheel speed of each drive wheel of the shooting device.

[0111] (4) Generate motion control commands based on the desired wheel speed set.

[0112] The error set is calculated based on the actual location, actual area, desired location, and desired area, and specifically includes: (1) Using the actual position and the desired position as input, the horizontal mapping error is calculated using the first error calculation formula.

[0113] (2) Using the actual area and the expected area as input, the distance error is calculated using the second error calculation formula.

[0114] (3) Combine the horizontal composition error and the distance error into an error set.

[0115] The error set includes horizontal mapping error and distance error. Using this error set as input, a PID controller is used to determine the desired speed set, which specifically includes: (1) Using the horizontal plotting error as input, the actual horizontal plotting error is calculated using the dead zone function.

[0116] (2) Using the actual horizontal mapping error as input, the expected lateral translation speed and the expected rotational speed are calculated by using a PID controller.

[0117] (3) Using the distance error as input, the actual distance error is calculated using the dead zone function.

[0118] (4) Using the actual distance error as input, the desired longitudinal forward and backward speed is calculated using a PID controller.

[0119] (5) The expected lateral translation speed, the expected rotational angular velocity and the expected longitudinal forward and backward speed are combined to form the expected speed set.

[0120] The desired velocity set is composed of the desired lateral translation velocity, the desired rotational angular velocity, and the desired longitudinal forward and backward velocity, specifically including: (1) Using the expected lateral translation speed, expected rotational angular velocity and expected longitudinal forward and backward speed as inputs, determine whether there is a collision risk based on the obstacle-occupied grid map.

[0121] (2) If not, the expected lateral translation speed, the expected rotational angular velocity and the expected longitudinal forward and backward speed are combined to form the expected speed set.

[0122] (3) If so, adjust the expected lateral translation speed, expected rotation speed and expected longitudinal advance speed respectively to obtain the adjusted expected lateral translation speed, adjusted expected rotation speed and adjusted expected longitudinal advance speed. Use the adjusted expected lateral translation speed, adjusted expected rotation speed and adjusted expected longitudinal advance speed as the expected lateral translation speed, expected rotation speed and expected longitudinal advance speed for the next iteration, and return to the step of "using the expected lateral translation speed, expected rotation speed and expected longitudinal advance speed as input, and determining whether there is a collision risk based on the obstacle-occupied grid map".

[0123] The desired velocity set includes the desired lateral translational velocity, the desired rotational angular velocity, and the desired longitudinal forward and backward velocity. With four drive wheels, the desired wheel speed set is calculated based on this set, specifically including: (1) Based on the expected velocity set, perform inverse kinematics to obtain the expected wheel speed of the first drive wheel, the expected wheel speed of the second drive wheel, the expected wheel speed of the third drive wheel and the expected wheel speed of the fourth drive wheel.

[0124] (2) The expected wheel speeds of the first drive wheel, the second drive wheel, the third drive wheel, and the fourth drive wheel are combined to form a set of expected wheel speeds.

[0125] This embodiment provides an automatic tracking control method based on embodied intelligence architecture and multimodal aesthetic evaluation. This method breaks away from the passive and rigid centering tracking logic of related technologies, and integrates visual and voice multimodal perception, 3D spatial environment cognition, and aesthetic decision-making driven by a large language model. It realizes active composition and embodied interactive tracking by the shooting device. Compared with automatic tracking solutions based on traditional visual servoing or hard-coded logic, this embodiment has the following significant advantages: (1) Possesses active aesthetic composition capabilities, significantly enhancing the artistry and visual appeal of video shooting. Existing technologies can only passively lock the geometric center of a face or body frame to the exact center of the video frame. Regardless of the subject's posture or direction of movement, the image remains unchanged, resulting in extremely rigid compositions and a lack of "cinematic feel." This embodiment can perform "active composition" like a professional human photographer. For example, when the subject's gaze moves to the right, it automatically positions the subject in the left third of the frame, leaving visual guidance space in front; when the subject needs to show their entire outfit, it automatically moves back to a specific framing. This advantage stems from the aesthetic reasoning and JSON actioncode output of the large language model in the multimodal aesthetic evaluation and decision-making module. By internalizing photographic aesthetic rules into the large language model, the rigid center coordinates are replaced with dynamically calculated expected anchor points, fundamentally changing the camera's tracking logic.

[0126] (2) It achieves deep semantic understanding and two-way “embodied” interactive experience, greatly reducing the threshold for single-person creation. Commercially available shooting devices typically rely on rigid preset gestures (such as making an OK sign to follow the camera, raising a hand to stop) or physical remote controls for interaction. These methods fail to understand the user's ambiguous intentions and operate in a one-way manner, unable to provide shooting feedback. In this embodiment, the user can free their hands and communicate with the shooting device using natural language (such as "The frame is too crowded," or "Move a little to the left"). The shooting device not only understands and executes these commands but also proactively analyzes the user's micro-expressions and skeletal posture, providing reverse voice guidance (such as playing "Please keep smiling," or "Relax your posture"). This advantage stems from multimodal acquisition and interaction devices (visual acquisition unit and voice interaction unit), structured scene description and Prompt engineering, and a two-way voice interaction closed loop. By introducing the powerful semantic reasoning capabilities of large language models such as Whisper speech recognition and LLaMA, combined with posture estimation and expression estimation, the barriers to human-computer interaction are completely broken down, forming a two-way closed loop of listening, seeing, moving, and speaking.

[0127] (3) Improve the robustness of target tracking in complex dynamic scenes (enhance the ability to resist occlusion and prevent loss) Most existing tracking algorithms rely on color histograms or simple single-frame target detection. However, if the target is facing away from the camera, multiple people are involved, or the target is briefly obscured by trees or passersby, the camera is prone to "identity jumps," causing the camera to either follow the passerby or lose focus altogether, resulting in camera erratic movements. This embodiment, however, can reliably track specific targets even in extremely complex environments (such as crowded anime conventions or streets with numerous obstacles), improving tracking accuracy and stability. This advantage stems from a multi-target tracking algorithm based on the fusion of appearance and motion features. By integrating Kalman filtering for kinematic trajectory prediction with Re-ID (person re-identification) appearance feature comparison, even if the target is completely obscured, the algorithm's trajectory memory and similarity matrix (Hungarian matching) can accurately recapture the target the moment it reappears.

[0128] (4) Improve the smoothness of physical space movement and the safety of active obstacle avoidance (shake prevention and safety) Because the visual detection frame inevitably experiences pixel-level jitter frame by frame, traditional shooting equipment that directly uses visual deviation as control input will cause high-frequency motor operation, resulting in severe image vibration. Furthermore, some wheeled shooting equipment lacks depth perception, making it prone to collisions with obstacles when following a subject backward. This embodiment improves the smoothness and stability of chassis movement, enhancing environmental safety and reducing collision risks when the shooting equipment follows a subject in physical space, whether lateral or backward. Smoothness and stability are achieved through dead-zone control and smoothing filtering algorithms, implementing an anti-shake filtering mechanism. By setting a dead-zone tolerance threshold to filter high-frequency noise, and supplemented by incremental PID smoothing control, the continuity of speed commands is improved. Smooth omnidirectional camera movement is achieved through inverse kinematics of the Mecanum wheel chassis, allowing the shooting equipment to slide laterally without changing the lens orientation. Safety is ensured by a 3D octree map containing depth information. The depth camera updates the logarithmic probability of voxel node occupancy in real time, and a 3D spatial interference check is performed before outputting the physical displacement velocity vector, strictly limiting motion planning to an unobstructed drivable area.

[0129] This application also provides an application scenario in which the above-described automatic tracking control method based on embodied intelligence architecture is applied. Specifically, the automatic tracking control method based on embodied intelligence architecture provided in this embodiment can be applied in video shooting scenarios. A video shooting scenario includes a control phase and a shooting phase. The control phase is used to issue motion control commands and voice output commands. The shooting phase is used to shoot the subject after the shooting device has moved into position based on the motion control commands and the subject has adjusted according to the voice output commands. The automatic tracking control method based on embodied intelligence architecture provided in this embodiment belongs to the control phase.

[0130] In one exemplary embodiment, an automatic tracking control system based on an embodied intelligence architecture is provided. The automatic tracking control system based on an embodied intelligence architecture includes: an image acquisition unit, a voice acquisition unit, a control unit, and a voice playback unit. The image acquisition unit, the voice acquisition unit, and the voice playback unit are all installed on the shooting device, and the image acquisition unit, the voice acquisition unit, and the voice playback unit are all communicatively connected to the control unit.

[0131] The image acquisition unit is used to acquire RGB images and depth images.

[0132] The voice acquisition unit is used to acquire voice input commands.

[0133] The control unit is used to execute the above-mentioned automatic tracking control method based on embodied intelligence architecture, and obtain motion control commands and voice output commands. The motion control commands are used to control the movement of the shooting device so that the human body to be shot is in the desired position of the shooting frame and occupies the desired area of ​​the shooting frame. The voice output commands are used to provide shooting guidance for the human body to be shot.

[0134] The voice playback unit is used to play voice output commands.

[0135] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements an automatic tracking control method based on an embodied intelligence architecture.

[0136] Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0138] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0139] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.

[0141] Any references to memory, databases, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0142] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0143] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An automatic tracking control method based on an embodied intelligence architecture, characterized in that, The automatic tracking control method based on embodied intelligence architecture includes: Before photographing the human body, acquire RGB images, depth images, and voice input commands; the RGB images and depth images are images obtained by photographing the environment in which the human body is located, and the voice input commands are voices issued by the human body to express the shooting needs; Target tracking is performed on the RGB image to determine the detection box corresponding to the human body to be photographed. Based on the detection box corresponding to the human body to be photographed, the true position, true area, human pose category, human face orientation, and human facial expression of the human body to be photographed are determined. Construct an obstacle occupancy grid map based on depth images; The voice input command is converted into text to obtain the input text; Using real location, real area, human pose category, human face orientation, human face expression, obstacle occupancy grid map and input text as input, a large language model is used to determine the expected position and expected area of ​​the human body to be photographed and the output text; the large language model is used to make camera movement decisions based on photographic aesthetics. Based on the actual position, actual area, desired position, and desired area, motion control commands are generated. The output text is then converted into speech to obtain speech output commands. The motion control commands are used to control the movement of the shooting device so that the human body to be shot is in the desired position of the shooting frame and occupies the desired area of ​​the shooting frame. The speech output commands are used to provide shooting guidance for the human body to be shot and complete the shooting.

2. The automatic tracking control method based on embodied intelligence architecture according to claim 1, characterized in that, Target tracking is performed on the RGB image to determine the detection box corresponding to the human body to be photographed, specifically including: Using an RGB image as input, a target detection model is used to determine a set of detection boxes; the set of detection boxes includes the detection box corresponding to each candidate target in the RGB image. Using the detection box corresponding to each candidate target in the RGB image as input, the appearance feature vector corresponding to each candidate target in the RGB image is determined by the appearance feature extraction model, and the appearance feature vectors corresponding to each candidate target in the RGB image are combined into a feature vector set. Based on the historical trajectory set, the state vector corresponding to the human body to be photographed in the RGB image is determined by the Kalman filter; the state vector includes the predicted detection box corresponding to the human body to be photographed. Based on the set of detection boxes, the set of feature vectors, the set of historical trajectories, and the state vector, the Hungarian algorithm is used to perform bipartite graph matching to obtain the detection boxes corresponding to the human body to be photographed.

3. The automatic tracking control method based on embodied intelligence architecture according to claim 1, characterized in that, Based on the detection bounding box corresponding to the human body to be photographed, the true position, true area, human posture category, human face orientation, and human facial expression of the human body are determined, specifically including: Based on the detection frame corresponding to the human body to be photographed, the true position and true area of ​​the human body to be photographed are determined; Using the detection box corresponding to the human body to be photographed as input, the coordinates of the human body key points are determined by the human body pose estimation model, and the human body pose category and human face orientation are determined based on the coordinates of the human body key points. Using the detection box corresponding to the human body to be photographed as input, the facial expression of the human body to be photographed is determined by the emotion classification model. The human pose estimation model uses an improved YOLOv8-pose, which is obtained by adding a CBAM attention mechanism module after each C2f module in the backbone network of YOLOv8-pose.

4. The automatic tracking control method based on embodied intelligence architecture according to claim 1, characterized in that, Based on the actual position, actual area, desired position, and desired area, motion control commands are generated, specifically including: Based on the actual location, actual area, desired location, and desired area, an error set is calculated. Using the error set as input, the desired speed set is determined by a PID controller. Based on the expected speed set, the expected wheel speed set is calculated; the expected wheel speed set includes the expected wheel speed of each drive wheel of the shooting device; Motion control commands are generated based on the desired wheel speed set.

5. The automatic tracking control method based on embodied intelligence architecture according to claim 4, characterized in that, Based on the actual location, actual area, desired location, and desired area, an error set is calculated, specifically including: Using the actual position and the desired position as input, the horizontal mapping error is calculated using the first error calculation formula. Using the actual area and the expected area as inputs, the distance error is calculated using the second error calculation formula. The horizontal composition error and the distance error are combined into an error set; The formula for calculating the first error is as follows: ； in, This refers to horizontal composition error; The x-coordinate of the actual location; Let x be the x-coordinate of the desired position. The width of the RGB image. The percentage of the x-coordinate of the desired position; The formula for calculating the second error is: ； in, This refers to distance error; This represents the actual area. For the desired area, The height of the RGB image. This represents the expected percentage.

6. The automatic tracking control method based on embodied intelligence architecture according to claim 4, characterized in that, The error set includes horizontal mapping error and distance error. Using this error set as input, a PID controller is used to determine the desired speed set, which specifically includes: Using the horizontal mapping error as input, the actual horizontal mapping error is calculated using the dead zone function; Using the actual horizontal mapping error as input, the desired lateral translation speed and desired rotational angular velocity are calculated using a PID controller. Using the distance error as input, the actual distance error is calculated using the dead zone function; Using the actual distance error as input, the desired longitudinal forward and backward speed is calculated using a PID controller. The desired lateral translation speed, the desired rotational angular velocity, and the desired longitudinal forward and backward speed are combined into a set of desired velocities; The dead zone function is: ； in, This represents the actual error. For error, when For horizontal composition error, then For actual horizontal composition error, when For distance error, then This represents the actual distance error. This represents the dead zone tolerance threshold corresponding to the error. It is a symbolic function.

7. The automatic tracking control method based on embodied intelligence architecture according to claim 6, characterized in that, The desired velocities are composed of the desired lateral translation velocity, the desired rotation angular velocity, and the desired longitudinal forward and backward velocity, specifically including: Using the expected lateral translation speed, expected rotational angular velocity, and expected longitudinal forward and backward speed as inputs, the existence of collision risk is determined based on the obstacle-occupied grid map. If not, then the expected lateral translation speed, the expected rotational angular velocity, and the expected longitudinal forward and backward speed will be combined into the expected speed set; If so, adjust the expected lateral translation speed, expected rotation angular velocity, and expected longitudinal advance / retreat speed respectively to obtain the adjusted expected lateral translation speed, adjusted expected rotation angular velocity, and adjusted expected longitudinal advance / retreat speed. Use the adjusted expected lateral translation speed, adjusted expected rotation angular velocity, and adjusted expected longitudinal advance / retreat speed as the expected lateral translation speed, expected rotation angular velocity, and expected longitudinal advance / retreat speed for the next iteration, and return to the step of "using the expected lateral translation speed, expected rotation angular velocity, and expected longitudinal advance / retreat speed as input to determine whether there is a collision risk based on the obstacle-occupied grid map".

8. The automatic tracking control method based on embodied intelligence architecture according to claim 4, characterized in that, The desired velocity set includes the desired lateral translational velocity, the desired rotational angular velocity, and the desired longitudinal forward and backward velocity. With four drive wheels, the desired wheel speed set is calculated based on this set, specifically including: Based on the desired velocity set, the inverse kinematics solution is performed to obtain the desired wheel speeds of the first drive wheel, the second drive wheel, the third drive wheel, and the fourth drive wheel. The expected wheel speeds of the first drive wheel, the second drive wheel, the third drive wheel, and the fourth drive wheel are combined to form a set of expected wheel speeds. The formula for calculating the desired wheel speed is as follows: ； ； ； ； in, The desired wheel speed of the first drive wheel; The radius of the drive wheel; The desired longitudinal forward and backward speed; The desired lateral translation speed; The desired angular velocity of rotation; It is half of the longitudinal wheelbase; It is half the lateral track width; The desired wheel speed of the second drive wheel; The desired wheel speed of the third drive wheel; This represents the desired wheel speed of the fourth drive wheel.

9. An automatic tracking and shooting control system based on an embodied intelligence architecture, characterized in that, The automatic tracking control system based on the embodied intelligent architecture includes: an image acquisition unit, a voice acquisition unit, a control unit, and a voice playback unit. The image acquisition unit, the voice acquisition unit, and the voice playback unit are all installed on the shooting equipment, and the image acquisition unit, the voice acquisition unit, and the voice playback unit are all communicatively connected to the control unit. The image acquisition unit is used to acquire RGB images and depth images; The voice acquisition unit is used to acquire voice input commands; The control unit is used to execute the automatic follow-up shooting control method based on the embodied intelligent architecture as described in any one of claims 1-8, and obtain motion control instructions and voice output instructions; the motion control instructions are used to control the movement of the shooting device so that the human body to be shot is in the desired position of the shooting frame and occupies the desired area of ​​the shooting frame, and the voice output instructions are used to provide shooting guidance for the human body to be shot. The voice playback unit is used to play voice output commands.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the automatic tracking control method based on the embodied intelligence architecture as described in any one of claims 1-8.

Images