Automatic following shooting system and method based on artificial intelligence

Abstract

The invention relates to an automatic following shooting system and method based on artificial intelligence. The automatic following shooting system based on artificial intelligence comprises a visual collector used for capturing images in real time; the audio collector is used for collecting voice information of an interpreter; the intelligent following shot decision-making device is used for completing feature extraction in face recognition, posture recognition, gesture recognition and voice semantic recognition, and calculating and outputting information including shooting positions, angles and focal lengths through a refined intelligent model; the device comprises a shooting controller, a mechanical arm and a camera, the shooting controller is connected with the mechanical arm and a camera body through data links and controls the mechanical arm and the camera body, and the shooting controller controls the shooting angle of the camera by controlling movement and steering of the mechanical arm and controls the focusing position of the camera by controlling the focal length and white balance of the camera. The automatic follow-up shooting system based on artificial intelligence is not limited by sites and teams, does not depend on the state of an operator, and meets the real-time requirement.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and photographing equipment, and in particular to an automatic tracking photography system and method based on artificial intelligence. Background Art

[0002] With the need for product promotion and display, photography technology has developed rapidly and been widely used.

[0003] The camera techniques, focal length switching, and timing judgment in the actual shooting process are highly dependent on professionals, and the output effect depends on the operator's experience and actual status. In actual situations, multiple lenses are often required to work simultaneously as alternatives, and the lens control center selects the appropriate camera position lens output information as the actual output; moreover, when multiple lenses exist at the same time, multiple shooting personnel and one or more center controllers are required to cooperate to complete it. In summary, the main problems with existing shooting technologies are: the output effect depends on the operator's experience and status, and there is a lack of guarantee for stable output; multiple shooting-related operators are required to work simultaneously, which is very unfriendly to small teams and low-cost projects.

[0004] Therefore, there is an urgent need to develop an intelligent automatic tracking photography solution that is easy to operate, has stable output, and is low in cost, which can provide a more easy-to-operate shooting solution without reducing the scope of actual application scenarios. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the output effect of the existing shooting technology depends on the operator's experience and status, lacks the guarantee of stable output, and requires multiple shooting-related operators to work at the same time, which is very unfriendly to small teams and low-cost projects.

[0006] In order to solve the above technical problems, according to one aspect of the present invention, an automatic tracking photography system based on artificial intelligence is provided, which includes: a visual collector, a camera for capturing images in real time, and using the captured images for face recognition, posture recognition or gesture recognition; an audio collector, which is used to collect the voice information of the tour guide, and can be a microphone; an intelligent tracking decision maker, which receives the information collected by the visual collector and the audio collector, completes the feature extraction in face recognition, posture recognition, gesture recognition, and speech semantic recognition, and calculates and outputs information including shooting position, angle and focal length through the refined intelligent model; a shooting controller, which receives the information collected by the visual collector and the audio collector, and completes the feature extraction in face recognition, posture recognition, gesture recognition, and speech semantic recognition, and calculates and outputs information including shooting position, angle and focal length through the extracted intelligent model; and a shooting controller, which receives the information collected by the visual collector and the audio collector, and uses the information collected by the audio collector to collect the voice information of the tour guide. The intelligent tracking decision maker outputs information including shooting position, angle and focal length. The shooting controller is connected to and controlled by the mechanical arm and the camera body through a data link. The shooting controller controls the shooting angle of the camera by controlling the movement and steering of the mechanical arm, and controls the focus position of the camera by controlling the focal length and white balance of the camera. The mechanical arm is set on a base and connected to the camera to support the camera and control and adjust the position and shooting angle of the camera. The camera is a camera device used for actual shooting in real scenes. The switch, focal length adjustment and white balance adjustment of the camera are controlled by the shooting controller.

[0007] According to an embodiment of the present invention, the shooting controller may include: a big data model trainer, which is used to establish an intelligent computing model and train it through data generated during actual use, and output information including the best shooting position, angle, focal length, and lens movement trajectory for the shooting controller to control the robotic arm and camera.

[0008] According to an embodiment of the present invention, the training model that can be used by the big data model trainer is based on an artificial intelligence algorithm based on deep learning. In order to achieve high-precision tracking and decision-making for face recognition, posture and gesture recognition, and speech recognition, the following models are used: face recognition, using convolutional neural network CNN for face feature extraction and recognition. The model FaceNet is used for face detection and tracking; posture and gesture recognition, using the model OpenPose based on the combination of convolutional neural network and recurrent neural network RNN, to identify human posture and gesture from video frames; speech recognition and semantic understanding, using the pre-trained speech model DeepSpeech to convert audio signals into understandable semantic information, thereby assisting intelligent tracking.

[0009] According to an embodiment of the present invention, a big data model trainer may adopt multimodal fusion, and through a multimodal data fusion algorithm, visual information including face, posture, and gesture is combined with audio information including voice and semantics to form a more comprehensive input data set. These multimodal inputs are processed through the multi-head attention mechanism in the improved Transformer model, thereby optimizing the accuracy and response speed of tracking decisions.

[0010] According to an embodiment of the present invention, a big data model trainer may adopt real-time tracking optimization and adopt the YOLO (You Only Look Once) real-time target detection algorithm to reduce the amount of calculation and improve the reasoning speed of the model, ensure rapid response in dynamic scenes, so as to improve the performance of real-time response and improve the existing target tracking algorithm; at the same time, Kalman filtering or particle filtering technology is adopted to optimize the prediction of the target and reduce the problem of camera motion lag.

[0011] According to an embodiment of the present invention, when the shooting controller optimizes the focal length and shooting angle, the focal length and angle can be optimized through a reinforcement learning algorithm, and a reinforcement learning agent can be trained to adjust the focal length and angle of the camera in real time during the shooting process, with the goal of maximizing the clarity of the photographed object and the stability of the position within the frame; through a reward mechanism, reinforcement learning can gradually learn the optimal shooting decision based on the quality of the shooting effect.

[0012] According to an embodiment of the present invention, the intelligent tracking decision maker may adopt a lightweight parameter model to meet the real-time requirements of photography tracking.

[0013] According to an embodiment of the present invention, the big data model trainer can train the artificial intelligence model by using the visual and audio information input collected by the visual collector and the audio collector and the actual camera movement video as samples.

[0014] According to an embodiment of the present invention, the audio collector will capture visual and audio information in real time, which can be provided as input information to the intelligent tracking decision maker to obtain information including the best shooting position, angle, focal length, and lens movement trajectory in real time.

[0015] According to another aspect of the present invention, there is provided an automatic tracking photography method based on artificial intelligence, which is implemented based on the automatic tracking photography system based on artificial intelligence as described above, and the automatic tracking photography method based on artificial intelligence comprises the following steps: step S1, visual acquisition, a camera that captures images in real time, and uses the captured images for face recognition, posture recognition or gesture recognition; step S2, audio acquisition, collecting voice information of the tour guide; step S3, intelligent tracking decision-making, receiving information collected by the visual collector and the audio collector, completing feature extraction in face recognition, posture recognition, gesture recognition, and speech semantic recognition, and calculating and outputting information including shooting position, angle and focal length through the refined intelligent model; conducting big data model training, establishing an intelligent computing model, and through actual The system is trained with data generated during actual use, and outputs information including the best shooting position, angle, focal length, and lens movement trajectory for the shooting controller to control the mechanical arm and the camera; step S4, shooting control, receiving information including the shooting position, angle, and focal length, the shooting controller is connected to the mechanical arm and the camera body through a data link and controls them, the shooting controller controls the shooting angle of the camera by controlling the movement and steering of the mechanical arm, and controls the focus position of the camera by controlling the focal length and white balance of the camera; wherein the mechanical arm is arranged on a base and connected to the camera, and is used to support the camera, and control and adjust the position and shooting angle of the camera; the camera is a camera device used for actual shooting in real scenes, and the switch, focal length adjustment, and white balance adjustment of the camera are controlled by the shooting controller.

[0016] Compared with the prior art, the technical solution provided by the embodiments of the present invention can at least achieve the following beneficial effects:

[0017] The automatic tracking photography system and method based on artificial intelligence of the present invention can use a visual collector and an audio collector as input modules, and the implementation conditions are simple and easy to implement.

[0018] According to the artificial intelligence-based automatic tracking photography system and method of the present invention, the artificial intelligence model can be trained through a large amount of existing video information containing camera techniques, and the amount of data that can be involved in the training is large; the amount of information collected is small, which is convenient for implementing a small model strategy and meeting the real-time requirements of implementation.

[0019] The automatic tracking photography system and method based on artificial intelligence of the present invention are convenient in that they can be implemented conveniently through visual and audio acquisition without being restricted by the site and the team; they are stable in that they can obtain stable tracking results through training with a large amount of existing data and do not rely on the status of specific operators; they are real-time in that they can meet the real-time requirements by importing a small amount of data and building a small model for intelligent calculation; they have broad application prospects in the commodity display and live broadcast market and great market potential. With the rapid development of commodity promotion, especially the personal live broadcast industry, the demand for low-cost, small-scale video production is very strong, and the present invention has great development potential and commercial value in the market.

[0020] The present invention provides an automatic tracking photography solution based on artificial intelligence. Through the convenience of implementation and output stability of artificial intelligence, it can effectively solve the limitations of traditional solutions and meet the easy implementation requirements of product promotion and anchor sales. It has low manpower requirements and is suitable for low-cost use by small-scale teams, and can be widely used. In the future, with the further evolution of technology and the expansion of the market, the present invention is expected to be widely used and promoted worldwide. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present invention, but are not intended to limit the present invention.

[0022] Figure 1 is a block diagram showing an automatic tracking photography system based on artificial intelligence according to an embodiment of the present invention;

[0023] Figure 2 is a flow chart showing an automatic tracking photography method based on artificial intelligence according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Unless otherwise defined, the technical or scientific terms used herein shall have the common meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the patent application specification and claims of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "a" do not indicate a quantity limitation, but indicate the existence of at least one.

[0026] Figure 1 is a block diagram showing an automatic tracking photography system based on artificial intelligence according to an embodiment of the present invention.

[0027] like Figure 1 As shown, the automatic tracking photography system based on artificial intelligence includes: a visual collector, an audio collector, an intelligent tracking decision maker, a shooting controller, a robotic arm and a camera.

[0028] The visual collector is a camera used to capture images in real time, and the captured images are used for face recognition, posture recognition or gesture recognition.

[0029] The audio collector is used to collect the voice information of the tour guide and can be a microphone.

[0030] The intelligent tracking decision maker receives information collected by the visual collector and the audio collector, completes feature extraction in face recognition, posture recognition, gesture recognition, and speech semantic recognition, and calculates and outputs information including shooting position, angle, and focal length through the refined intelligent model.

[0031] The shooting controller receives information including shooting position, angle and focal length output by the intelligent tracking decision maker. The shooting controller is connected to and controlled by the robotic arm and the camera body through a data link. The shooting controller controls the shooting angle of the camera by controlling the movement and steering of the robotic arm, and controls the focus position of the camera by controlling the focal length and white balance of the camera.

[0032] The mechanical arm is set on the base and connected to the camera to support the camera and control and adjust the position and shooting angle of the camera. The camera is a camera device used for actual shooting in real scenes. The camera's switch, focus adjustment, and white balance adjustment are controlled by the shooting controller.

[0033] The automatic tracking photography system and method based on artificial intelligence of the present invention can use a visual collector and an audio collector as input modules, and the implementation conditions are simple and easy to implement.

[0034] According to one or some embodiments of the present invention, the shooting controller includes: a big data model trainer. The big data model trainer is used to establish an intelligent computing model, and train it through data generated during actual use, and output information including the best shooting position, angle, focal length, and lens motion trajectory for the shooting controller to control the mechanical arm and the camera.

[0035] According to the artificial intelligence-based automatic tracking photography system and method of the present invention, the artificial intelligence model can be trained through a large amount of existing video information containing camera techniques, and the amount of data that can be involved in the training is large; the amount of information collected is small, which is convenient for implementing a small model strategy and meeting the real-time requirements of implementation.

[0036] According to one or some embodiments of the present invention, the training model used by the big data model trainer is based on an artificial intelligence algorithm based on deep learning. In order to achieve high-precision tracking and decision-making for face recognition, posture and gesture recognition, and speech recognition, the following model is used:

[0037] Face recognition, using convolutional neural network (CNN) to extract and recognize facial features. The FaceNet model is used for face detection and tracking.

[0038] Posture and gesture recognition, using OpenPose, a model based on a combination of convolutional neural networks and recurrent neural networks (RNN), to recognize people's postures and gestures from video frames.

[0039] Speech recognition and semantic understanding use the pre-trained speech model DeepSpeech to convert audio signals into understandable semantic information, thereby assisting intelligent tracking.

[0040] According to one or some embodiments of the present invention, the big data model trainer adopts multimodal fusion, and through the multimodal data fusion algorithm, combines visual information including face, posture, and gesture with audio information containing voice and semantics to form a more comprehensive input data set. These multimodal inputs are processed by the multi-head attention mechanism in the improved Transformer model, thereby optimizing the accuracy and response speed of tracking decisions.

[0041] According to one or some embodiments of the present invention, the big data model trainer adopts real-time tracking optimization and adopts the YOLO (You Only Look Once) real-time target detection algorithm to reduce the amount of calculation and improve the reasoning speed of the model, ensure rapid response in dynamic scenes, so as to improve the performance of real-time response and improve the existing target tracking algorithm; at the same time, Kalman filtering or particle filtering technology is used to optimize the prediction of the target and reduce the problem of camera motion lag.

[0042] According to one or some embodiments of the present invention, when the shooting controller optimizes the focal length and shooting angle, the focal length and angle are optimized through a reinforcement learning algorithm, and a reinforcement learning agent is trained to adjust the focal length and angle of the camera in real time during the shooting process, with the goal of maximizing the clarity of the photographed object and the stability of the position within the frame; through a reward mechanism, reinforcement learning can gradually learn the optimal shooting decision based on the quality of the shooting effect.

[0043] According to one or some embodiments of the present invention, the intelligent tracking decision maker adopts a lightweight parameter model to meet the real-time requirements of photography tracking.

[0044] According to one or some embodiments of the present invention, the big data model trainer trains the artificial intelligence model by using the visual and audio information input collected by the visual collector and the audio collector and the actual camera movement video as samples.

[0045] According to one or some embodiments of the present invention, the audio collector captures visual and audio information in real time and provides it as input information to the intelligent tracking decision maker to obtain information including the best shooting position, angle, focal length, and lens motion trajectory in real time.

[0046] Figure 2 is a flow chart showing an automatic tracking photography method based on artificial intelligence according to an embodiment of the present invention.

[0047] The automatic tracking photography system and method based on artificial intelligence of the present invention are convenient in that they can be implemented conveniently through visual and audio acquisition without being restricted by the site and the team; they are stable in that they can obtain stable tracking results through training with a large amount of existing data and do not rely on the status of specific operators; they are real-time in that they can meet the real-time requirements by importing a small amount of data and building a small model for intelligent calculation; they have broad application prospects in the commodity display and live broadcast market and great market potential. With the rapid development of commodity promotion, especially the personal live broadcast industry, the demand for low-cost, small-scale video production is very strong, and the present invention has great development potential and commercial value in the market.

[0048] like Figure 2 As shown, the automatic tracking photography method based on artificial intelligence includes the following steps:

[0049] Step S1, visual acquisition, a camera that captures images in real time, and uses the captured images for face recognition, posture recognition or gesture recognition.

[0050] Step S2: audio collection, collecting the voice information of the tour guide.

[0051] Step S3, intelligent follow-up decision, receiving information collected by the visual collector and the audio collector, completing feature extraction in face recognition, posture recognition, hand gesture recognition, and speech semantic recognition, and calculating and outputting information including shooting position, angle, and focal length through the refined intelligent model; conducting big data model training, establishing an intelligent computing model, and training with data generated during actual use, and outputting information including the best shooting position, angle, focal length, and lens motion trajectory for the shooting controller to control the robotic arm and the camera.

[0052] Step S4, shooting control, receiving information including shooting position, angle and focal length, the shooting controller is connected to the mechanical arm and the camera body through a data link and controls the shooting angle of the camera by controlling the movement and steering of the mechanical arm, and controls the focus position of the camera by controlling the focal length and white balance of the camera. The mechanical arm is set on the base and connected to the camera, used to support the camera, and control and adjust the position and shooting angle of the camera; the camera is a camera device used for actual shooting in real scenes, and the camera's switch, focal length adjustment, and white balance adjustment are controlled by the shooting controller.

[0053] The present invention provides an automatic tracking photography solution based on artificial intelligence. Through the convenience of implementation and output stability of artificial intelligence, it can effectively solve the limitations of traditional solutions and meet the easy implementation requirements of product promotion and anchor sales. It has low manpower requirements and is suitable for low-cost use by small-scale teams, and can be widely used. In the future, with the further evolution of technology and the expansion of the market, the present invention is expected to be widely used and promoted worldwide.

[0054] The above description is merely an exemplary embodiment of the present invention and is not intended to limit the protection scope of the present invention. The protection scope of the present invention is determined by the appended claims.

Claims

1. An automatic tracking photography system based on artificial intelligence, comprising: A visual collector is a camera used to capture images in real time and use the captured images for face recognition, posture recognition or gesture recognition; Audio collector, used to collect the voice information of the tour guide; An intelligent follow-up decision maker receives information collected by the visual collector and the audio collector, completes feature extraction in face recognition, posture recognition, gesture recognition, and speech semantic recognition, and calculates and outputs information including shooting position, angle, and focal length through the extracted intelligent model; A shooting controller receives information including shooting position, angle and focal length output by the intelligent follow-up decision maker. The shooting controller is connected to and controls the mechanical arm and the camera body through a data link. The shooting controller controls the shooting angle of the camera by controlling the movement and steering of the mechanical arm, and controls the focus position of the camera by controlling the focal length and white balance of the camera. A mechanical arm, which is arranged on a base and connected to the camera, and is used to support the camera and control and adjust the position and shooting angle of the camera; The camera is a camera device used for actual shooting in real scenes. The camera's switch, focus adjustment, and white balance adjustment are controlled by the shooting controller.

2. The automatic tracking photography system based on artificial intelligence as claimed in claim 1, wherein: The shooting controller comprises: A big data model trainer is used to establish an intelligent computing model and train it through data generated during actual use, and output information including the best shooting position, angle, focal length, and lens movement trajectory for the shooting controller to control the robotic arm and camera.

3. The automatic tracking photography system based on artificial intelligence as claimed in claim 2, wherein: The training model used by the big data model trainer is based on the deep learning artificial intelligence algorithm. In order to achieve high-precision tracking and decision-making for face recognition, posture and gesture recognition, and voice recognition, the following model is used: Face recognition, using convolutional neural network (CNN) to extract and recognize facial features. The FaceNet model is used for face detection and tracking; Posture and gesture recognition, using OpenPose, a model based on a combination of convolutional neural networks and recurrent neural networks (RNN), to recognize people's postures and gestures from video frames; Speech recognition and semantic understanding use the pre-trained speech model DeepSpeech to convert audio signals into understandable semantic information, thereby assisting intelligent tracking.

4. The automatic tracking photography system based on artificial intelligence as claimed in claim 2, wherein: The big data model trainer adopts multimodal fusion. Through the multimodal data fusion algorithm, it combines visual information including face, posture, and gesture with audio information containing voice and semantics to form a more comprehensive input data set. These multimodal inputs are processed through the multi-head attention mechanism in the improved Transformer model, thereby optimizing the accuracy and response speed of tracking decisions.

5. The automatic tracking photography system based on artificial intelligence as claimed in claim 2, wherein: The big data model trainer uses real-time tracking optimization and the YOLO real-time target detection algorithm to reduce the amount of calculation and improve the model's reasoning speed, ensuring rapid response in dynamic scenes to improve real-time response performance; at the same time, Kalman filtering or particle filtering technology is used to optimize target prediction and reduce the problem of camera motion lag.

6. The automatic tracking photography system based on artificial intelligence as claimed in claim 1, wherein: When the shooting controller optimizes the focal length and shooting angle, the focal length and angle are optimized by using a reinforcement learning algorithm, and a reinforcement learning agent is trained to adjust the focal length and angle of the camera in real time during the shooting process, with the goal of maximizing the clarity of the object being shot and the stability of the position within the frame; Through the reward mechanism, reinforcement learning can gradually learn the optimal shooting decisions based on the quality of the shooting effects.

7. The automatic tracking photography system based on artificial intelligence as claimed in claim 1, wherein: The intelligent tracking decision maker adopts a lightweight parameter model to meet the real-time requirements of photography tracking.

8. The automatic tracking photography system based on artificial intelligence as claimed in claim 2, wherein: The big data model trainer trains the artificial intelligence model by using the visual and audio information input collected by the visual collector and the audio collector and the actual camera movement video as samples.

9. The automatic tracking photography system based on artificial intelligence as claimed in claim 1, wherein: The audio collector will capture visual and audio information in real time and provide it as input information to the intelligent tracking decision maker to obtain information including the best shooting position, angle, focal length, and lens movement trajectory in real time.

10. An automatic tracking photography method based on artificial intelligence, the automatic tracking photography method is implemented based on the automatic tracking photography system based on artificial intelligence according to any one of claims 1 to 9, and the automatic tracking photography method based on artificial intelligence comprises the following steps: Step S1, visual acquisition, a camera that captures images in real time, and uses the captured images for face recognition, posture recognition or gesture recognition; Step S2: audio collection, collecting the voice information of the narrator; Step S3, intelligent follow-up decision, receiving the information collected by the visual collector and the audio collector, completing the feature extraction in face recognition, posture recognition, gesture recognition, and speech semantic recognition, and calculating and outputting the information including shooting position, angle and focal length through the refined intelligent model; conducting big data model training, establishing an intelligent computing model, and training through the data generated during actual use, and outputting the information including the best shooting position, angle, focal length, and lens motion trajectory for the shooting controller to control the mechanical arm and the camera; Step S4, shooting control, receiving information including shooting position, angle and focal length, the shooting controller is connected to the mechanical arm and the camera body through a data link and controls them, the shooting controller controls the shooting angle of the camera by controlling the movement and steering of the mechanical arm, and controls the focus position of the camera by controlling the focal length and white balance of the camera; wherein, The mechanical arm is arranged on a base and connected to a camera, and is used to support the camera and control and adjust the position and shooting angle of the camera; the camera is a camera device used for actual shooting in real scenes, and the switch, focus adjustment and white balance adjustment of the camera are controlled by the shooting controller.

Images