Automatic stage lighting control method and system based on machine vision

Abstract

The invention discloses a stage lighting automatic control method based on machine vision, and relates to the field of automatic control, and the method comprises the steps: obtaining a human body dynamic video; identifying a target performer in the stage performers, and marking a target key frame; constructing and training a human body action recognition model; constructing a key skeleton time sequence diagram network of the target performer according to the target key frame; acquiring human body position information of the target performer according to the human body dynamic video; extracting human body action information by using the human body action recognition model; and generating a light control strategy in combination with the human body action information and the human body position information. According to the invention, high-efficiency, accurate and low-cost stage lighting automatic control on the stage can be effectively realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of automatic control, and in particular to a method and system for automatic control of stage lighting based on machine vision. Background Art

[0002] The lighting design and control of stage art has always been one of the key factors for the success of a performance. By adjusting parameters such as light brightness, color, and angle, various specific visual effects can be created, which can greatly improve the quality and viewing experience of stage performances. Therefore, stage lighting is an indispensable part of stage performances.

[0003] Most of the existing stage lighting systems rely on manual control. Lighting technicians need to manually adjust the lights according to the specific content of the performance. For complex stage settings and rapidly changing performance rhythms, it is difficult to keep up with the rhythm of the stage performance through manual control, and there is a situation where lighting adjustments are not timely. In addition, there are many types of large-scale stage lights, so the number of people to control the lights also needs to be increased, which greatly increases labor costs. In addition, due to the increase in the number of lighting technicians, the difficulty of coordination between lighting technicians has also increased, greatly increasing the risk of operational errors. Summary of the invention

[0004] The embodiments of the present application provide a method and system for automatic stage lighting control based on machine vision, which are used to solve the problem that the prior art cannot perform fast, low-cost and high-precision all-round control of large stage lighting.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions: In a first aspect, a method for automatic stage lighting control based on machine vision is provided, the method comprising: The human body dynamic video of the stage performers is obtained by using image acquisition devices arranged around the target stage; Using a target detection algorithm to identify a target performer among the stage performers from key frames in the human body dynamic video, and marking the key frame where the target performer is located as a target key frame; Constructing a human action recognition model including a human feature extraction module and a human feature weighting module, and completing model training of the human action recognition model through a pre-constructed behavior recognition training set; Extracting key skeleton points of the target performer from each of the target key frames, calculating human skeleton data based on the key skeleton points, and constructing a key skeleton timing graph network by combining the key skeleton points of all the target key frames; According to the human body dynamic video and using a Kalman filter algorithm, the human body position information of the target performer is acquired in real time; Inputting the key skeleton timing graph network and the human skeleton data into the trained human motion recognition model to obtain human motion information; A lighting control strategy is generated in real time by combining the human body motion information and the human body position information.

[0006] Optionally, the step of using a target detection algorithm to identify a target performer among the stage performers from key frames in the human body dynamic video comprises the following steps: For any key frame in the human body dynamic video, use a target detection algorithm to identify and select the stage performer in the key frame to obtain a key image area; If the number of the key image regions is greater than 1, then for any of the key image regions, the RGB color space of the key image region is converted into the HSV color space, and a clustering algorithm is used to identify the body color features of the stage performers in the key image region; Gray-scaling the key image area to obtain a gray-scale image area; Using an edge detection algorithm to remove the image background in the grayscale image area, retaining the human body image area in the grayscale image area, and performing filtering processing on the human body image area; Performing corrosion and expansion operations on the filtered human body image region according to morphology to obtain a complete human body image region; Calculating the human body morphological features of the stage performer based on the complete human body image area; Acquire a pre-stored reference color feature and a reference morphological feature of a target actor; comparing the human body color feature and the human body morphological feature with the reference color feature and the reference morphological feature respectively, and if the human body color feature is the same as the reference color feature, and the human body morphological feature is the same as the reference morphological feature, marking the stage performer in the key image area as a target performer; If the number of key image regions is equal to 1, the stage performer in the key image region is marked as a target performer.

[0007] Optionally, constructing a human action recognition model including a human feature extraction module and a human feature weighting module, and performing model training on the human action recognition model using a pre-constructed behavior recognition training set comprises the following steps: Combining the multi-stream network and convolutional neural network model to build a multi-stream feature extraction submodule; A graph feature extraction submodule is constructed based on the graph convolutional neural network model, and a human feature extraction module is obtained by combining the graph feature extraction submodule and the multi-stream feature extraction submodule; A human feature weighting module is constructed based on a self-attention mechanism, and a human action recognition model is constructed by combining the human feature extraction module, the human feature weighting module and a preset feature fusion module; The pre-constructed behavior recognition training set is input into the human action recognition model to perform iterative training on the human action recognition model. When the number of iterative training reaches a preset maximum number of iterations, the training of the human action recognition model is completed.

[0008] Optionally, extracting the key skeleton points of the target performer from each of the target key frames, calculating the human skeleton data according to the key skeleton points, and constructing the key skeleton timing graph network by combining the key skeleton points of all the target key frames comprises the following steps: For any of the target key frames in the human body dynamic video, a human body posture estimation algorithm is used to perform single-person posture estimation on the target performer in the target key frame to obtain a bone point heat map of the target performer; Calculating the skeleton point confidence and the two-dimensional coordinates of the skeleton point according to the skeleton point heat map, extracting the skeleton points of the target performer according to the skeleton point confidence, and obtaining a plurality of key skeleton points of the target performer; Generate a skeleton diagram of the target performer according to the key skeleton points; Calculate the human skeleton data of the target performer according to the two-dimensional coordinates of the skeleton points and the skeleton diagram; The key bone points corresponding to the skeleton graphs of adjacent frames are connected by time edges to obtain a key bone timing graph network of the human body dynamic video.

[0009] Optionally, the step of calculating the human skeleton data of the target performer according to the two-dimensional coordinates of the skeleton points and the skeleton diagram comprises the following steps: Acquire parameter information of the image acquisition device, and obtain depth information of the key skeleton points by combining the parameter information and the two-dimensional coordinates of all the skeleton points; Calculate the three-dimensional coordinates of all the key skeleton points by combining the depth information and the two-dimensional coordinates of the skeleton points to obtain a plurality of skeleton point coordinates; Calculate the skeleton vector between two connected key skeleton points according to the skeleton diagram and through the skeleton point coordinates; Calculate the bone angle between the connected key bones according to the bone point coordinates, wherein the key bone is a bone connecting two key bone points; Calculate the coordinate changes between the key skeleton points of adjacent frames according to the skeleton point coordinates to obtain skeleton motion data; The skeleton point coordinates, the skeleton vectors, the skeleton angles and the skeleton motion data are integrated to obtain the human skeleton data of the target performer.

[0010] Optionally, the step of acquiring the human body position information of the target performer in real time according to the human body dynamic video and using a Kalman filter algorithm comprises the following steps: The first target key frame in the time sequence of the human body dynamic video is used as the initial key frame, and the last target key frame is used as the cutoff key frame; In the initial key frame, determining the first initial position information of the target performer according to the skeleton point coordinates and using a position determination algorithm; Using a Kalman filter algorithm to predict the predicted position information of the next target key frame of the initial key frame according to the first initial position information; In a target key frame next to the initial key frame, determining the second initial position information of the target performer according to the skeleton point coordinates and using the position determination algorithm; The second initial position information is modified according to the predicted position information to obtain the body position information of the target performer in the next target key frame of the initial key frame; The next target key frame of the initial key frame is used as a new initial key frame, and the above position prediction and updating steps are repeatedly performed until the cutoff key frame is used as the initial key frame and the repetition is stopped.

[0011] Optionally, the location determination algorithm comprises the following steps: Combining all the skeleton point coordinates to construct a minimum bounding box for the target performer, and using the geometric center of the minimum bounding box as the body center point of the target performer; Filter out the coordinates of the bone points within a preset range around the center point of the human body to obtain one or more core bone point coordinates; If the number of the core skeleton point coordinates is equal to one, the core skeleton point coordinates are used as the initial position information of the target performer; If the number of the core skeleton point coordinates is greater than one, the average value of all the core skeleton point coordinates is calculated to obtain the average coordinates, and the average coordinates are used as the initial position information of the target performer.

[0012] Optionally, the step of inputting the key skeleton timing graph network and the human skeleton data into the trained human motion recognition model to obtain the human motion information comprises the following steps: Filling the human skeleton data with data, and aligning the timestamps of the human skeleton data after completing the data filling with the human body dynamic video; Inputting the human skeleton data after the timestamp alignment into the multi-stream feature extraction submodule in the human motion recognition model, extracting data features from the human skeleton data through the multi-stream feature extraction submodule to obtain skeleton point features, skeleton vector features, skeleton angle features and skeleton motion features; Performing feature fusion on the bone point features, the bone vector features, the bone angle features and the bone motion features to obtain multi-stream data features; Inputting the key skeleton timing graph network into the graph feature extraction submodule of the human action recognition model to perform feature extraction through graph convolution operation to obtain skeleton graph features; Inputting the multi-stream data features and the skeleton graph features into the human feature weighting module of the human action recognition model to allocate attention to the multi-stream data features and the skeleton graph features, and performing attention weight allocation on the multi-stream data features and the skeleton graph features according to the attention; Inputting the multi-stream data features and the skeleton graph features after the attention weight allocation into a preset multi-dimensional feature fusion module for multi-dimensional feature fusion to obtain human behavior features; The human behavior characteristics are classified using a preset classification function to obtain an action category probability distribution, and human action information is obtained according to the action category probability distribution.

[0013] In a second aspect, the present application provides a machine-readable storage medium, characterized in that the machine-readable storage medium stores instructions, and the instructions are used to enable a machine to execute a method for automatic stage lighting control based on machine vision according to any one of the first aspects.

[0014] In a third aspect, the present application provides a stage lighting automatic control system based on machine vision, characterized in that it includes: a memory configured to store instructions; and The processor is configured to call the instruction from the memory and implement a method for automatic stage lighting control based on machine vision according to any one of the first aspects when executing the instruction.

[0015] Through the above technical scheme, the target performer among the stage performers is identified and marked by the human body dynamic video and the target detection algorithm, which is convenient for the subsequent action recognition and coordinate positioning of the target performer. After the target performer is marked, a human body action recognition model including a human body feature extraction module and a human body feature weighting module is constructed, and the human body action information of the target performer is efficiently and accurately identified by the human body action recognition model. The target key frame containing the target performer in the human body dynamic video is extracted by key bone points and human body bone data, and the key bone point extraction is used to construct a key bone timing graph network, which is convenient for the subsequent use of the human body action recognition model to extract multi-dimensional features of the target performer, that is, the multi-stream data features of the human body bone data and the bone graph features of the key bone timing graph network are extracted respectively, and the human body action information of the target performer can be more accurately identified to facilitate the subsequent design of the lighting control strategy. The real-time positioning of the target performer is realized by the Kalman filter algorithm, which is convenient for the light tracking of the target performer, improves the audience's viewing experience, and enhances the stage effect. In summary, the present application provides a method for automatic stage lighting control that can be performed on the stage efficiently, accurately and at low cost, which greatly improves the stage performance effect.

[0016] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of a method for automatic stage lighting control based on machine vision provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a human action recognition model provided in an embodiment of the present application; Figure 3 A key skeleton timing diagram network is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0019] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0020] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0021] Figure 1 The flowchart of a method for automatic stage lighting control based on machine vision according to an embodiment of the present application is schematically shown. Figure 1 As shown, the embodiment of the present application provides a method for automatic control of stage lighting based on machine vision, which may include the following steps: S101. Acquire a human body dynamic video of a stage performer through image acquisition devices arranged around a target stage.

[0022] In this embodiment, the image acquisition device mainly includes a high-definition camera, and a position that can capture the entire target stage is pre-arranged around the target stage, such as a position in the center in front of the target stage. The image acquisition device such as a high-definition camera is used to capture the stage performers (such as drama actors) performing on the target stage, and a performance video containing the stage performers, that is, a human body dynamic video, is obtained.

[0023] S102: using a target detection algorithm to identify a target performer among the stage performers from the key frames in the human body dynamic video, and marking the key frame where the target performer is located as a target key frame.

[0024] In this embodiment, the target detection algorithm is a core technology in the field of computer vision, which aims to identify and locate one or more specific object instances from an image or video. Commonly used target detection algorithms include the R-CNN series, the YOLO series, etc. Taking YOLO (target detection algorithm based on deep learning) as an example, the key frame is input into the neural network, and the key frame is divided into multiple grids through operations such as convolution pooling. Each grid predicts whether there is a stage performer in the grid, so as to select the stage performer and obtain the key image area. If there are multiple key image areas, the human color features of the key image area are extracted, the key image area is grayed and the background is removed to obtain the human image area, the human morphological features are extracted from the human image area, the target performer among the stage performers is identified according to the human color features and the human morphological features, and the key frame containing the target performer is marked as the target key frame, so as to facilitate the subsequent extraction of the key bone points of the target performer.

[0025] The step of identifying the target performer is because in some large-scale performances, there are multiple stage performers on the target stage. Generally, in order to improve the stage effect, the core performers on the stage, such as the leading actors in a stage play, the lead singers in a concert, etc., will be highlighted and tracked. Therefore, it is necessary to identify the core performers, that is, the target performers, in advance and design special lighting control strategies for the core performers in real time to enhance the audience's viewing experience.

[0026] S103, constructing a human action recognition model including a human feature extraction module and a human feature weighting module, and completing model training of the human action recognition model through a pre-constructed behavior recognition training set.

[0027] In this embodiment, the human feature extraction module includes a multi-stream feature extraction submodule and a graph feature extraction submodule. The multi-stream feature extraction submodule is constructed based on a multi-stream network and a convolutional neural network model. The multi-stream network refers to a network that inputs data streams of multiple modes into different branch networks and fuses them using a multi-stream fusion mechanism, thereby utilizing the advantages of different data streams to improve the accuracy and performance of the algorithm. The convolutional neural network model is a type of feedforward neural network that includes convolution calculations and has a deep structure. The multi-stream feature extraction submodule can extract the multi-dimensional skeletal features of the target performer and improve the accuracy of the human action recognition model in identifying human type information. The graph feature extraction module is constructed based on a graph convolutional neural network model. The graph convolutional neural network model can perform convolution operations on graph structure data through spectral and spatial domain methods to extract the features of the graph structure data. The human feature weighting module is constructed based on a self-attention mechanism. The self-attention mechanism pays more attention to more discriminative features by learning attention masks, assigns weights to features of different dimensions, and improves the performance of the human action recognition model. The pre-built action recognition training set contains performance videos of the same type as the target stage performance, such as stage play videos with the same script, dance performance videos of the same dance, etc.

[0028] S104, extracting the key skeleton points of the target performer from each target key frame respectively, and calculating the human skeleton data based on the key skeleton points, and constructing a key skeleton timing graph network by combining the key skeleton points of all target key frames.

[0029] In this embodiment, the human body posture estimation algorithm is used to extract the key skeleton points of the target performer in the target key frame, and the skeleton point heat map of the target performer is obtained. The skeleton point heat map can show the probability data of each pixel in the target key frame being a key skeleton point. According to the skeleton point heat map, the two-dimensional coordinates of each pixel, i.e., the two-dimensional coordinates of the skeleton point, can be extracted. The skeleton point confidence of each pixel is calculated according to the probability data, and the pixel points with confidence greater than the preset confidence threshold are screened out to obtain the key skeleton points. The same method is used to extract the key skeleton points of all target key frames. For the key skeleton points in each target key frame, the key skeleton points can be directly connected according to the human body structure to obtain the skeleton diagram of the target performer. The key skeleton points located in the same part of the human body in adjacent target key frames are connected by time, for example, the left shoulder skeleton point of the first target key frame is connected with the left shoulder skeleton point of the second target key frame, and the key skeleton points in all different target key frames are connected one by one to obtain the key skeleton timing diagram network of the human body dynamic video. The key bone timing graph network includes the information of the target performer in the spatial dimension and the temporal dimension, and can reflect the action characteristics of the target performer at a certain time node. The human skeleton data includes the coordinates of the bone points, the bone vectors, the bone angles and the bone movement data. The human skeleton data can reflect the action characteristics of the target performer from the data level. Combining the key bone timing graph network and the human skeleton data can more accurately reflect the action characteristics of the target performer, making the recognition results of the human action recognition model more accurate and reliable.

[0030] S105, obtaining the body position information of the target performer in real time according to the body dynamic video and using the Kalman filter algorithm.

[0031] In this embodiment, the initial position information of the target performer in the initial key frame is calculated according to the coordinates of the key skeleton points, and then the Kalman filter algorithm is used to predict the predicted position information of the next frame of the initial key frame according to the initial position information, and the calculated initial position information of the next frame of the initial key frame is corrected using the predicted position information, and then the Kalman filter algorithm is continued to be used to continuously correct the position information calculated by the coordinates of the key skeleton points to obtain more accurate human body position information. The Kalman filter algorithm is an algorithm that uses the linear system state equation to optimally estimate the system state by inputting and outputting observation data of the system. Since the observation data includes the influence of noise and interference in the system, the optimal estimate can also be regarded as a filtering process. The Kalman filter algorithm includes two steps: prediction and analysis: in the prediction stage, the predicted value of the mode state at the current moment is generated according to the mode state at the previous moment; in the analysis stage, the observation data is introduced, and the mode state is re-analyzed using the minimum variance estimation method. For example, if the initial position information at time a is known, the predicted position information at time a+1 can be predicted using the Kalman filter. Real-time acquisition of the target performer's body position information is to ensure that the stage lighting can continuously track the target performer. The target performer is the core figure of the target stage. The target performer can be highlighted through lighting, making the audience more engaged in watching the performance.

[0032] S106, inputting the key skeleton timing graph network and human skeleton data into the trained human motion recognition model to obtain human motion information.

[0033] In this embodiment, the key skeleton timing graph network is input into the graph feature extraction submodule to extract the graph feature, and the skeleton graph feature is obtained. The human skeleton data is input into the multi-stream feature extraction submodule for data feature extraction to obtain the skeleton point feature, the skeleton vector feature, the skeleton angle feature and the skeleton motion feature. The skeleton point feature, the skeleton vector feature, the skeleton angle feature and the skeleton motion feature are subjected to feature fusion to obtain the multi-stream data feature. The skeleton graph feature and the multi-stream data feature are weighted by the human feature weighting module in the human motion recognition model. After completing the weight assignment, the skeleton graph feature and the multi-stream data feature are subjected to feature fusion to obtain the human behavior feature. The human behavior feature is subjected to feature classification to obtain the action category probability distribution, and the human motion information is obtained according to the action category probability distribution. The action category probability distribution refers to the identified action category probability, and the action category with the largest probability is used as the human motion information. The human motion information refers to the action category of the target performer. The action category of the target performer is identified to set the local angle of the stage lighting, to ensure that the target performer is fully illuminated by the stage lighting, so that the audience can more clearly watch the performance details of the target performer and enhance the viewing experience.

[0034] S107, generating a lighting control strategy in real time by combining the human body motion information and the human body position information.

[0035] In this embodiment, the local angle of the stage light is set according to the human body movement information to ensure that the performer is fully illuminated by the light, and the global angle and light position of the stage light can be set according to the human body position information to ensure that the stage light can illuminate the performer.

[0036] For example, during a concert, if the target performer sings solo in the center of the target stage, the lights above the center of the target stage need to be turned on to illuminate the target performer. At the same time, the lights on the left and right sides of the stage will immediately adjust their angles to illuminate both sides of the target performer, ensuring that the target performer is covered by the stage lights. After the lighting control strategy is generated, the lighting control strategy is converted into a lighting control instruction, the lighting control instruction is encoded, and the obtained encoding information is sent to the pre-set lighting control system, and the lighting control system performs real-time lighting control on the target stage according to the encoding information.

[0037] In one embodiment, using a target detection algorithm to identify a target performer among stage performers from key frames in a human motion video comprises the following steps: For any key frame in the human body dynamic video, the target detection algorithm is used to identify and select the stage performer in the key frame to obtain the key image area; If the number of key image regions is greater than 1, then for any key image region, the RGB color space of the key image region is converted into the HSV color space, and the human body color features of the stage performers in the key image region are identified using a clustering algorithm; Gray-scale the key image area to obtain a gray-scale image area; Using edge detection algorithm to remove the image background in the grayscale image area, retaining the human image area in the grayscale image area, and performing filtering on the human image area; According to morphology, the human body image region after filtering is subjected to corrosion and expansion operations to obtain a complete human body image region; The human morphological features of the stage performer are calculated based on the complete human image area; Acquire a pre-stored reference color feature and a reference morphological feature of a target actor; Comparing the human body color feature and the human body morphology feature with the reference color feature and the reference morphology feature respectively, if the human body color feature is the same as the reference color feature, and the human body morphology feature is the same as the reference morphology feature, then marking the stage performer in the key image area as the target performer; If the number of key image regions is equal to 1, the stage performer in the key image region is marked as the target performer.

[0038] In this embodiment, key frames that may contain target performers in the human body dynamic video are first extracted. For example, deep learning is used to count the number of heads in the human body dynamic video as the number of stage performers, and multiple frames of images before and after the change in the number of stage performers are used as key frames. Multiple frames of images at the beginning and end of the human body dynamic video are also selected as key frames. By extracting key frames, computing power can be greatly saved and recognition efficiency can be improved. After obtaining the key frames of the human body dynamic video, the target performer is identified and framed in each key frame using a target detection algorithm. Common target detection algorithms include two-stage methods based on deep learning represented by the R-CNN series and single-stage methods based on deep learning represented by the YOLO series. Taking the YOLO series as an example, the steps for using YOLO for target detection are: adjust the key frame to a fixed size, such as 640x640 pixels, extract the local features of the key frame and reduce its dimension through multiple convolutional layers and pooling layers, and then divide the key frame into multiple grid units. Each grid unit predicts multiple bounding boxes. Each bounding box contains 5 prediction values, namely the position of the center point of the bounding box in the grid, the confidence, the width and height of the bounding box. The confidence indicates whether there is a target in the bounding box. Each bounding box also predicts the category probability of the target. These probabilities are multiplied by the confidence to obtain the final target probability, that is, the probability that the target selected in the bounding box is a stage performer. The bounding boxes with probabilities greater than the preset probability threshold are screened out to obtain the key image area containing the stage performer.

[0039] In some large stages, there are multiple performers on the stage. In order to improve the stage effect, it is necessary to design a special lighting control strategy for the core performers (i.e., the target performers) among the stage performers. For example, there are dancers and lead singers in a concert. The lead singer is the core performer, and the lead singer needs to be highlighted and tracked to ensure the quality of the concert. Therefore, it is necessary to identify the target performer from multiple stage performers.

[0040] Specifically, each key image area is first converted to a color space, and the RGB (red, green, and blue) color space is converted to the HSV (hue, saturation, and brightness) color space. The RGB value is normalized to a range, and then the hue, saturation, and brightness are calculated based on R, G, and B to complete the color space conversion. After the color space conversion is completed, the main color in the key image area is identified using a clustering algorithm to obtain the body color features of the stage performer. A commonly used clustering algorithm can be the k-means (mean clustering) algorithm, which clusters the pixels in the key image area, groups pixels of similar colors into the same class, and calculates the center of each class, i.e., the main color, extracts the center color of each class, and calculates the frequency of occurrence of each color. The color with the highest frequency of occurrence is taken as the main color, i.e., the body color features of the stage performer. This is because in general, the clothing color of the target performer is different from that of other stage performers, so the body color features are used as one of the factors for selecting the target performer.

[0041] The key image area is grayed by using the displacement method, average method, weighted average method and other methods, which can make the outline of the stage performer in the key image area clearer and the details more obvious. After obtaining multiple gray image areas, the edge point set of the gray image area is extracted by the edge detection algorithm, and the obtained edge point set is integrated into the edge feature of the gray image area to obtain the outline feature of the stage performer. The background of the gray image area is segmented according to the outline feature to obtain the human image area in the gray image area. The human image area refers to the area containing the body of the stage performer. Commonly used edge detection algorithms include Sobel operator, Laplace operator and Roberts operator. Taking Sobel operator as an example, the edge is identified by calculating the gradient of the gray value of the image pixel to obtain the edge point. After that, the human body image area is filtered. Median filtering can be used to remove noise in the image to make the human body image area smoother. Morphological processing is performed on the human body image area after median filtering. The corrosion operation can eliminate isolated and small points in the image and smooth rough edges. The dilation operation can fill the holes caused by the corrosion operation. The human body image area is made more complete through morphology, and a complete character image area that can more accurately reflect the body shape characteristics of the stage performer is obtained.

[0042] Obtain the parameters of the image acquisition device, including the focal length of the lens, the size of the lens photosensitive element, etc., and refer to the size of the stage props, such as the microphone, pillars, steps, etc. on the stage, and then calculate the human morphological features of all stage performers based on the pre-complete character image area. The human morphological features include the height, width and length of the human outline. Obtain the pre-stored human color features and human morphological features of the target performer, and use them as the reference color features and reference morphological features to compare with the human color features and human morphological features of all stage performers. If there is a stage performer in a key image area whose human color features and human morphological features are the same as the reference color features and reference morphological features, then mark the stage performer in the key image area as the target performer.

[0043] In addition, in some cases, there is only one stage performer on the stage from beginning to end, such as a talk show stage, so the number of key image areas obtained is one, and the stage performer in the key image area is directly marked as the target performer.

[0044] In one embodiment, constructing a human action recognition model including a human feature extraction module and a human feature weighting module, and training the human action recognition model using a pre-constructed behavior recognition training set comprises the following steps: Combining the multi-stream network and convolutional neural network model to build a multi-stream feature extraction submodule; A graph feature extraction submodule is constructed based on the graph convolutional neural network model, and a human feature extraction module is obtained by combining the graph feature extraction submodule and the multi-stream feature extraction submodule; A human feature weighting module is constructed based on the self-attention mechanism, and a human action recognition model is constructed by combining the human feature extraction module, the human feature weighting module and the preset feature fusion module. The pre-built behavior recognition training set is input into the human action recognition model to iteratively train the human action recognition model. When the number of iterative training reaches the preset maximum number of iterations, the training of the human action recognition model is completed.

[0045] In this embodiment, refer to Figure 2The human feature extraction module includes a multi-stream feature extraction submodule and a graph feature extraction submodule. The multi-stream feature extraction submodule is constructed based on a multi-stream network and a convolutional neural network model. A multi-stream network refers to a network that inputs data streams of multiple modes into different branch networks and fuses them using a multi-stream fusion mechanism, thereby utilizing the advantages of different data streams to improve the accuracy and performance of the algorithm. The convolutional neural network model is a type of feedforward neural network with a deep structure that includes convolution calculations. The multi-stream feature extraction submodule can extract the multi-dimensional skeletal features of the target performer and improve the accuracy of the human action recognition model in identifying human type information. The multi-stream network and the convolutional neural network model together constitute a multi-stream convolutional neural network model, which can extract features from different data in the human skeleton data. The multi-stream convolutional neural network model can use a combination of input layers, multi-layer convolutional layers, activation layers, pooling layers, and fully connected layers to extract local features of each data in the human skeleton data through convolution operations.

[0046] The graph feature extraction submodule is built based on the graph convolutional neural network model. The graph convolutional neural network model can perform convolution operations on graph structure data through spectral and spatial domain methods to extract the features of graph structure data. The basic structure of the graph convolutional neural network includes input layer, graph convolution layer, nonlinear activation function, graph pooling layer and output layer. The graph convolutional neural network model updates the key bone points by training the learning function and aggregating the neighbor information of each key bone point in the key bone time series graph network. The node features and structural features of the key bone time series graph network are extracted by aggregating the updated information multiple times. The human feature weighting module is built based on the self-attention mechanism. The self-attention mechanism pays more attention to more discriminative features by learning attention masks, and weights are assigned to features of different dimensions to improve the performance of the human action recognition model.

[0047] The pre-constructed behavior recognition training set contains performance videos of the same type as the target stage performance, such as stage play videos with the same script, dance performance videos with the same dance, etc. The pre-constructed behavior recognition training set is input into the human action recognition model, key skeleton points and human skeleton data are extracted from the videos in the training set, and a key skeleton timing graph network for model training is constructed. The key skeleton timing graph network for model training is input into the graph feature extraction submodule to extract graph features, and the human skeleton data for model training is input into the multi-stream feature extraction submodule. Local data features are extracted for each data in the human skeleton data, and multiple local data features extracted are subjected to feature fusion to obtain data features. The data features and graph features are input into the human feature weighting module for weight allocation. After the weight allocation is completed, feature fusion is performed, and the fused features obtained after feature fusion are classified into human action categories to obtain training action categories. The human action recognition model is trained multiple times, and the loss is calculated. The parameters of each module in the human action recognition model, such as the number of convolutional layers, the selection of activation functions, etc., are continuously adjusted until the preset maximum number of training times is reached and the model training is completed.

[0048] In one embodiment, extracting key skeleton points of a target performer from each target key frame, calculating human skeleton data based on the key skeleton points, and constructing a key skeleton timing graph network by combining the key skeleton points of all target key frames includes the following steps: For any target key frame in the human body dynamic video, a human body posture estimation algorithm is used to estimate the single person posture of the target performer in the target key frame to obtain the bone point heat map of the target performer; Calculate the skeleton point confidence and the two-dimensional coordinates of the skeleton point according to the skeleton point heat map, extract the skeleton points of the target performer according to the skeleton point confidence, and obtain multiple key skeleton points of the target performer; Generate a skeleton diagram of the target performer based on key bone points; Calculate the human skeleton data of the target performer according to the two-dimensional coordinates of the skeleton points and the skeleton diagram; The corresponding key bone points in the skeleton graphs of adjacent frames are temporally connected to obtain a key bone timing graph network for human body dynamic video.

[0049] In this embodiment, refer to Figure 3Commonly used single-person posture estimation algorithms include top-down posture estimation algorithms and bottom-up posture estimation algorithms. Taking the bottom-up posture estimation algorithm as an example, the process of the bottom-up multi-person posture estimation algorithm is to first detect all the bone joints in the picture, then cluster them, and assign the joints to different people, so as to obtain the final skeleton diagram. Specifically, the target performer in the target key frame is firstly subjected to bone point extraction to obtain the target performer's bone point heat map. The bone point heat map can show the probability data that each pixel in the target key frame is a key bone point. According to the bone point heat map, the two-dimensional coordinates of each pixel can be extracted to obtain the corresponding two-dimensional coordinates of the bone point. The bone point confidence of each pixel is calculated according to the probability data, and the pixel points with confidence greater than the preset confidence threshold are screened out to obtain the key bone points. The same method is used to extract the key bone points of all target key frames. The key bone points are the bone points of the key parts of the human body, such as the bone points of the shoulders, elbows, knees, etc. The key bone points are directly connected according to the human body structure to obtain the skeleton diagram of the target performer. The number of nodes in the skeleton graph of each target key frame is the number of corresponding key bone points, and the connecting edges are the bones between the key bone points. A skeleton graph can only represent the action information of one frame, and can only represent the action of the target performer at a certain time node. It cannot represent the action change information of the target performer at different time nodes. Therefore, it is necessary to connect the key bone points located in the same part of the human body between adjacent target key frames through time. For example, the left shoulder bone point of the first target key frame is connected to the left shoulder bone point of the second target key frame. After connecting the key bone points in all different target key frames one by one, the key bone timing graph network of the human body dynamic video is obtained. The time edge is a feature edge that contains time features. The key bone timing graph network can reflect the action changes of the target dancer in the time dimension and the space dimension.

[0050] After obtaining the two-dimensional coordinates of the skeleton points, the depth information of the key skeleton points is obtained using the monocular depth estimation algorithm according to the parameters of the image acquisition device, including the focal length and the position of the principal point. The depth information is used as the value of the Z axis in the three-dimensional coordinate system. The three-dimensional coordinate system takes the center point of the target stage as the origin. Then, the two-dimensional coordinates of the skeleton points are converted into the coordinates of the X axis and the Y axis in the three-dimensional coordinate system by combining the focal length and the position of the principal point, thereby obtaining the three-dimensional coordinates of the key skeleton points, namely the skeleton point coordinates. The basic principle of monocular depth estimation is to infer the depth information in the scene by analyzing the texture, shadow, occlusion and other information in the image.

[0051] In one embodiment, calculating the human skeleton data of the target performer according to the two-dimensional coordinates of the skeleton points and the skeleton diagram includes the following steps: Obtain parameter information of the image acquisition device, and combine the parameter information with the two-dimensional coordinates of all skeleton points to obtain depth information of key skeleton points; Combining the depth information and the two-dimensional coordinates of the skeleton points, the three-dimensional coordinates of all key skeleton points are calculated to obtain the coordinates of multiple skeleton points; Calculate the bone vector between two connected key bone points according to the skeleton graph and the bone point coordinates; Calculate the bone angles between the connected key bones according to the bone point coordinates, where the key bone is the bone connecting two key bone points; Calculate the coordinate changes between key bone points in adjacent frames according to the bone point coordinates to obtain bone motion data; The bone point coordinates, bone vectors, bone angles and bone motion data are integrated to obtain the human skeleton data of the target performer.

[0052] In this embodiment, according to the parameters of the image acquisition device, including the focal length and the position of the principal point, the depth information of the key skeleton points is obtained by using a monocular depth estimation algorithm, and the depth information is used as the value of the Z axis in a three-dimensional coordinate system, the three-dimensional coordinate system takes the center point of the target stage as the origin, and then the two-dimensional coordinates of the skeleton points are converted into the coordinates of the X axis and the Y axis in the three-dimensional coordinate system in combination with the focal length and the position of the principal point, thereby obtaining the three-dimensional coordinates of the key skeleton points, i.e., the skeleton point coordinates. The basic principle of monocular depth estimation is to infer the depth information in the scene by analyzing the texture, shadow, occlusion and other information in the image. The calculation formula of the skeleton point coordinates is as follows: ; ; ; in, is the bone point coordinate, is the 2D coordinate of the bone point, As the main point position, is the focal length, It is the depth information of key bone points.

[0053] The bone vector of the key bone can be calculated by calculating the coordinate difference between two key bone points. The key bone is the bone connecting two key bone points. The bone vector contains the length and direction information of the key bone, which can reflect the local characteristics between two key bone points. The bone angle can also be calculated based on the bone point coordinates and using the cosine theorem. The bone angle includes the angle of the two key bones on the plane constructed by the X-axis and the Y-axis, the angle on the plane constructed by the X-axis and the Z-axis, and the angle on the plane constructed by the Y-axis and the Z-axis in the three-dimensional coordinate system. The bone motion data is calculated by subtracting the coordinates of two key bone points connected by the time edge, which can reflect the target dancer's movement changes at different times. After obtaining the bone point coordinates, bone vectors, bone angles and bone motion data of all target key frames, they are integrated to obtain the human skeleton data of the target performer.

[0054] In one embodiment, obtaining the human body position information of the target performer in real time based on the human body dynamic video and using the Kalman filter algorithm includes the following steps: The first target key frame in the time sequence of the human body dynamic video is used as the initial key frame, and the last target key frame is used as the cutoff key frame; In the initial key frame, first initial position information of the target performer is determined according to the coordinates of the skeleton points and using a position determination algorithm; Using a Kalman filter algorithm to predict the predicted position information of the next target key frame of the initial key frame according to the first initial position information; In a target key frame next to the initial key frame, second initial position information of the target performer is determined according to the coordinates of the skeleton points and using a position determination algorithm; The second initial position information is modified according to the predicted position information to obtain the human body position information of the target performer in the next target key frame of the initial key frame; The next target key frame of the initial key frame is used as a new initial key frame, and the above position prediction update steps are repeated until the end key frame is used as the initial key frame.

[0055] In this embodiment, the target key frames are sorted according to the shooting time, the target key frame with the earliest shooting time is used as the initial key frame, and the target key frame obtained by the latest shooting is used as the cutoff key frame. The initial position information of the target performer is calculated according to the three-dimensional coordinates of the key skeleton points in the initial key frame. The initial position information refers to the three-dimensional coordinates of the target performer on the stage. In order to further improve the reliability of the initial position information, the initial position information is corrected using the Kalman filter algorithm. The Kalman filter algorithm is an optimal recursive filtering method under a discrete system. It can have a more accurate estimate of the dynamic system and can estimate the motion state of the motion tracking target at the next moment according to the state of the motion target at the previous moment.

[0056] The Kalman filter algorithm is used to predict the predicted position information of the next target key frame of the initial key frame based on the initial position information, and then the initial position information of the target performer in the next target key frame is calculated based on the skeletal point coordinates of the target performer in the next target key frame of the initial key frame. The initial position information of the next target key frame is corrected using the predicted position information of the next target key frame to obtain more accurate human body position information. The next target key frame of the initial key frame is used as the initial key frame to continue to perform the above steps, and the initial position information is continuously calculated based on the skeletal point coordinates, and the predicted position information is predicted based on the initial position information. The initial position information is used as the initial position information, and the initial position information is corrected using the predicted position information until the end key frame is used as the initial key frame, which means that the performance of the target performer is about to end, and there is no need to continue to predict its human body position information, and the cycle ends. Through the correction of the Kalman filter algorithm, the accuracy of obtaining human body position information can be improved.

[0057] In one embodiment, the location determination algorithm comprises the following steps: Combining all the bone point coordinates, constructing a minimum bounding box for the target performer, and taking the geometric center of the minimum bounding box as the center point of the target performer's body; Filter out the coordinates of the bone points within a preset range around the center point of the human body to obtain one or more core bone point coordinates; If the number of core skeleton point coordinates is equal to one, the core skeleton point coordinates are used as the initial position information of the target performer; If the number of core skeleton point coordinates is greater than one, the average value of all core skeleton point coordinates is calculated to obtain the average coordinates, and the average coordinates are used as the initial position information of the target performer.

[0058] In this embodiment, the minimum external box refers to the smallest cube or cuboid that can completely surround a set of points (or objects) in three-dimensional space, that is, the cube or cuboid that can wrap all key bone points. The method of constructing the minimum external box is to first find the maximum and minimum coordinates of all key bone points on the X-axis, Y-axis and Z-axis respectively, and obtain the maximum value coordinate and the minimum value coordinate according to the maximum and minimum coordinates on the X-axis, Y-axis and Z-axis. For example: A=(1,2,3), B=(5,0,6), C=(6,2,3), then the minimum value coordinate is (1,0,3), the maximum value coordinate is (6,2,6), and the straight line connecting the points corresponding to the maximum value coordinate and the minimum value coordinate is the diagonal of the minimum external box. The geometric center of the minimum external box is the center point of the diagonal, which is used as the human body center point of the target performer. The coordinates of the human body center point are the average of the maximum value coordinate and the minimum value coordinate. The coordinates of the bone points within the preset range around the human body center point are screened out to obtain one or more core bone point coordinates. The prediction range refers to the range of the sphere with the human body center point as the origin. If only one core bone point coordinate is screened out, the core bone point coordinate is directly used as the initial position information of the target performer. If multiple core bone point coordinates are screened out, the average value of all the core bone point coordinates screened out is calculated to obtain the average coordinate, which is used as the initial position information of the target performer.

[0059] This application also provides another method for calculating initial position information, comprising the following steps: The signal transmission time is obtained in real time through a plurality of signal receiving base stations pre-installed around the target stage and a signal transmitter pre-worn by the target performer; Obtaining base station location information of all signal receiving base stations; The initial position information of the target performer is calculated by combining the signal transmission time and the base station location information and using the principle of three-sided positioning.

[0060] In this embodiment, the UWB technology (ultra-wideband technology) can be used to transmit and receive signals using a signal receiving base station and a signal transmitter, and data can be transmitted through nanosecond non-sinusoidal narrow pulses to perform real-time positioning of the target performer on the target stage. First, the time required for signal transmission between all signal receiving base stations and signal transmitters is obtained, and the base station location information of the pre-arranged signal receiving base stations is obtained at the same time. The distance between the signal receiving base stations is calculated based on the base station location information, and then the initial position information of the target performer is calculated using the trilateral positioning principle. The calculation formula for the initial position information is as follows: ; ; in, is the initial position information of the target performer, , and is the base station location information of signal receiving base station 1, signal receiving base station 2 and signal receiving base station 3, , and is the distance between signal receiving base station 1, signal receiving base station 2 and signal receiving base station 3, It is the area of ​​the triangle formed by signal receiving base station 1, signal receiving base station 2 and signal receiving base station 3.

[0061] When the target stage is an extra-large stage and a large number of image acquisition devices are installed on it, if the initial position information of the target performer is obtained based on the coordinates of the skeleton points, the required amount of calculation is large. This is because the initial position information obtained from the human body dynamic video captured by different image acquisition devices may have deviations due to different shooting angles. Therefore, the different initial position information needs to be corrected. The more image acquisition devices there are, the more computing power is consumed. Therefore, the real-time transmission and reception of signals can be used to locate the target performer in real time, which can reduce the amount of calculation.

[0062] In one embodiment, inputting the key skeleton timing graph network and human skeleton data into a trained human motion recognition model to obtain human motion information includes the following steps: Filling the human skeleton data, and aligning the timestamps of the human skeleton data with the human body dynamic video after the data filling is completed; The human skeleton data after the timestamp alignment is input into the multi-stream feature extraction submodule in the human action recognition model, and the multi-stream feature extraction submodule is used to extract data features of the human skeleton data to obtain bone point features, bone vector features, bone angle features and bone motion features; Perform feature fusion on bone point features, bone vector features, bone angle features and bone motion features to obtain multi-stream data features; The key skeleton timing graph network is input into the graph feature extraction submodule of the human action recognition model to extract features through graph convolution operation to obtain skeleton graph features; Input the multi-stream data features and the skeleton graph features into the human feature weighting module of the human action recognition model to allocate attention to the multi-stream data features and the skeleton graph features, and allocate attention weights to the multi-stream data features and the skeleton graph features according to the attention; Input the multi-stream data features and skeleton graph features after attention weight allocation into the preset multi-dimensional feature fusion module for multi-dimensional feature fusion to obtain human behavior features; The human behavior characteristics are classified using a preset classification function to obtain the action category probability distribution, and the human action information is obtained based on the action category probability distribution.

[0063] In the present embodiment, data filling is performed on human skeleton data. This is because some frames are removed when selecting key frames, so some frames in the human body dynamic video do not have human skeleton data, so zero value filling is performed on these frames without human skeleton data. After completing the zero value filling, the human skeleton data and the human body dynamic video are time-stamp aligned, that is, they are adjusted to the same time axis, and the human skeleton data that completes the time stamp alignment is input into the multi-stream feature extraction submodule in the human motion recognition model. The multi-stream feature extraction module extracts features from the bone point coordinates, bone vectors, bone angles and bone motion data in the human skeleton data, respectively, to obtain bone point features, bone vector features, bone angle features and bone motion features. Specifically, the human skeleton data is input into the multi-stream feature extraction submodule, and all human skeleton data is first normalized, and then convolved through a multi-layer convolutional neural network (which can be a two-layer 1*1 convolutional neural network), and nonlinear factors are introduced through the activation function to perform feature dimension reduction. Finally, the features are output after the fully connected layer to obtain bone point features, bone vector features, bone angle features, and bone motion features. The bone point features, bone vector features, bone angle features, and bone motion features are fused to obtain multi-stream data features. Common feature fusion methods include splicing, pyramid pooling, addition, multiplication, and deconvolution. Taking the splicing method as an example, the bone point features, bone vector features, bone angle features, and bone motion features are spliced ​​into a long vector to obtain multi-stream data features.

[0064] The key skeleton timing graph network is input into the graph feature extraction submodule of the human motion recognition model, and temporal convolution and spatial convolution are performed alternately for several times (can be three times). The temporal convolution processes the changes of key skeletons in different target key frames of the key skeleton timing graph network to obtain the characteristics of the target performer's motion changes over time. The spatial convolution processes the connection relationship between key skeleton points in the same target key frame of the key skeleton timing graph network to obtain the motion characteristics of the target performer at a certain time node. Finally, the features of the two different dimensions are added together to obtain the skeleton graph features. The graph feature extraction submodule captures the co-occurrence collaborative representation of key skeleton points for spatial and temporal dynamic changes.

[0065] The human feature weighting module is constructed by the self-attention mechanism. The multi-stream data features and the skeleton graph features are input into the human feature weighting module. The multi-stream data features and the skeleton graph features are first vectorized and converted into a feature matrix. The feature matrix is ​​mapped to the query, key, and value vectors using linear changes. The dot product between the query vector and the key vector is calculated to obtain the attention score matrix. The dot product result is divided by a scaling factor (which can be the square root of the key vector) to stabilize the gradient. The scaled attention score matrix is ​​converted into a probability value between [0,1] using a normalization function to obtain the attention weight. The normalization function can be a softmax function. The obtained attention weight is multiplied by the value vector to complete the attention weight allocation for the multi-stream data features and the skeleton graph features. The multi-stream data features and the skeleton graph features after the weight allocation are input into the multi-dimensional feature fusion module for weighted summation to obtain the human behavior features. The human behavior features are classified to obtain the action category probability distribution, and the human action information is obtained according to the action category probability distribution. The probability distribution of action categories refers to the probability of the identified action categories, and the action category with the highest probability is taken as the human action information. Human action information refers to the action category of the target performer. The action category of the target performer is identified to set the local angle of the stage lighting to ensure that the target performer is fully illuminated by the stage lighting, so that the audience can more clearly see the performance details of the target performer and improve the viewing experience.

[0066] The present application also discloses a stage lighting automatic control system based on machine vision, which is characterized by comprising: a memory configured to store instructions; and A processor is configured to call instructions from a memory and implement any of the above-mentioned automatic stage lighting control methods based on machine vision when executing the instructions.

[0067] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0068] Among them, the memory can be an internal storage unit of a computer device, such as a hard disk or memory of a computer device, or an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD) or a flash memory card (FC) equipped on the computer device, and the memory can also be a combination of an internal storage unit and an external storage device of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or is to be output, and this application does not impose any restrictions on this.

[0069] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned method for automatic stage lighting control based on machine vision.

[0070] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0071] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0075] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0076] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0077] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0078] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for automatic stage lighting control based on machine vision, characterized in that: The method comprises the following steps: The human body dynamic video of the stage performers is obtained by using image acquisition devices arranged around the target stage; Using a target detection algorithm to identify a target performer among the stage performers from key frames in the human body dynamic video, and marking the key frame where the target performer is located as a target key frame; Constructing a human action recognition model including a human feature extraction module and a human feature weighting module, and completing model training of the human action recognition model through a pre-constructed behavior recognition training set; Extracting key skeleton points of the target performer from each of the target key frames, calculating human skeleton data based on the key skeleton points, and constructing a key skeleton timing graph network by combining the key skeleton points of all the target key frames; According to the human body dynamic video and using a Kalman filter algorithm, the human body position information of the target performer is acquired in real time; Inputting the key skeleton timing graph network and the human skeleton data into the trained human motion recognition model to obtain human motion information; A lighting control strategy is generated in real time by combining the human body motion information and the human body position information.

2. The method according to claim 1, characterized in that: The step of using a target detection algorithm to identify a target performer among the stage performers from key frames in the human body dynamic video comprises the following steps: For any key frame in the human body dynamic video, use a target detection algorithm to identify and select the stage performer in the key frame to obtain a key image area; If the number of the key image regions is greater than 1, then for any of the key image regions, the RGB color space of the key image region is converted into the HSV color space, and a clustering algorithm is used to identify the body color features of the stage performers in the key image region; Gray-scaling the key image area to obtain a gray-scale image area; Using an edge detection algorithm to remove the image background in the grayscale image area, retaining the human body image area in the grayscale image area, and performing filtering processing on the human body image area; Performing corrosion and expansion operations on the filtered human body image region according to morphology to obtain a complete human body image region; Calculating the body morphological features of the stage performer based on the complete body image area; Acquire a pre-stored reference color feature and a reference morphological feature of a target actor; comparing the human body color feature and the human body morphological feature with the reference color feature and the reference morphological feature respectively, and if the human body color feature is the same as the reference color feature, and the human body morphological feature is the same as the reference morphological feature, marking the stage performer in the key image area as a target performer; If the number of key image regions is equal to 1, the stage performer in the key image region is marked as a target performer.

3. The method according to claim 1, characterized in that The steps of constructing a human action recognition model including a human feature extraction module and a human feature weighting module, and training the human action recognition model using a pre-constructed behavior recognition training set include the following steps: Combining the multi-stream network and convolutional neural network model to build a multi-stream feature extraction submodule; A graph feature extraction submodule is constructed based on the graph convolutional neural network model, and a human feature extraction module is obtained by combining the graph feature extraction submodule and the multi-stream feature extraction submodule; A human feature weighting module is constructed based on a self-attention mechanism, and a human action recognition model is constructed by combining the human feature extraction module, the human feature weighting module and a preset feature fusion module; The pre-constructed behavior recognition training set is input into the human action recognition model to perform iterative training on the human action recognition model. When the number of iterative training reaches a preset maximum number of iterations, the training of the human action recognition model is completed.

4. The method according to claim 1, characterized in that: The steps of extracting the key skeleton points of the target performer from each of the target key frames, calculating the human skeleton data according to the key skeleton points, and constructing a key skeleton timing graph network by combining the key skeleton points of all the target key frames include the following steps: For any of the target key frames in the human body dynamic video, a human body posture estimation algorithm is used to perform single-person posture estimation on the target performer in the target key frame to obtain a bone point heat map of the target performer; Calculating the skeleton point confidence and the two-dimensional coordinates of the skeleton point according to the skeleton point heat map, extracting the skeleton points of the target performer according to the skeleton point confidence, and obtaining a plurality of key skeleton points of the target performer; Generate a skeleton diagram of the target performer according to the key skeleton points; Calculate the human skeleton data of the target performer according to the two-dimensional coordinates of the skeleton points and the skeleton diagram; The key bone points corresponding to the skeleton graphs of adjacent frames are connected by time edges to obtain a key bone timing graph network of the human body dynamic video.

5. The method according to claim 4, characterized in that The step of calculating the human skeleton data of the target performer according to the two-dimensional coordinates of the skeleton points and the skeleton diagram comprises the following steps: Acquire parameter information of the image acquisition device, and obtain depth information of the key skeleton points by combining the parameter information and the two-dimensional coordinates of all the skeleton points; Calculate the three-dimensional coordinates of all the key skeleton points by combining the depth information and the two-dimensional coordinates of the skeleton points to obtain a plurality of skeleton point coordinates; Calculate the skeleton vector between two connected key skeleton points according to the skeleton diagram and through the skeleton point coordinates; Calculate the bone angle between the connected key bones according to the bone point coordinates, wherein the key bone is a bone connecting two key bone points; Calculate the coordinate changes between the key skeleton points of adjacent frames according to the skeleton point coordinates to obtain skeleton motion data; The skeleton point coordinates, the skeleton vectors, the skeleton angles and the skeleton motion data are integrated to obtain the human skeleton data of the target performer.

6. The method according to claim 5, characterized in that The method of acquiring the human body position information of the target performer in real time according to the human body dynamic video and using the Kalman filter algorithm comprises the following steps: The first target key frame in the time sequence of the human body dynamic video is used as the initial key frame, and the last target key frame is used as the cutoff key frame; In the initial key frame, determining the first initial position information of the target performer according to the skeleton point coordinates and using a position determination algorithm; Using a Kalman filter algorithm to predict the predicted position information of the next target key frame of the initial key frame according to the first initial position information; In a target key frame next to the initial key frame, determining the second initial position information of the target performer according to the skeleton point coordinates and using the position determination algorithm; The second initial position information is modified according to the predicted position information to obtain the body position information of the target performer in the next target key frame of the initial key frame; The next target key frame of the initial key frame is used as a new initial key frame, and the above position prediction and updating steps are repeatedly performed until the cutoff key frame is used as the initial key frame and the repetition is stopped.

7. The method according to claim 6, characterized in that The position determination algorithm comprises the following steps: Combining all the skeleton point coordinates to construct a minimum bounding box for the target performer, and using the geometric center of the minimum bounding box as the body center point of the target performer; Filter out the coordinates of the bone points within a preset range around the center point of the human body to obtain one or more core bone point coordinates; If the number of the core skeleton point coordinates is equal to one, the core skeleton point coordinates are used as the initial position information of the target performer; If the number of the core skeleton point coordinates is greater than one, the average value of all the core skeleton point coordinates is calculated to obtain the average coordinates, and the average coordinates are used as the initial position information of the target performer.

8. The method according to claim 1, characterized in that The step of inputting the key skeleton timing graph network and the human skeleton data into the trained human motion recognition model to obtain human motion information comprises the following steps: Filling the human skeleton data with data, and aligning the timestamps of the human skeleton data after completing the data filling with the human body dynamic video; Inputting the human skeleton data after the timestamp alignment into the multi-stream feature extraction submodule in the human motion recognition model, extracting data features from the human skeleton data through the multi-stream feature extraction submodule to obtain skeleton point features, skeleton vector features, skeleton angle features and skeleton motion features; Performing feature fusion on the bone point features, the bone vector features, the bone angle features and the bone motion features to obtain multi-stream data features; Inputting the key skeleton timing graph network into the graph feature extraction submodule of the human action recognition model to perform feature extraction through graph convolution operation to obtain skeleton graph features; Inputting the multi-stream data features and the skeleton graph features into the human feature weighting module of the human action recognition model to allocate attention to the multi-stream data features and the skeleton graph features, and performing attention weight allocation on the multi-stream data features and the skeleton graph features according to the attention; Inputting the multi-stream data features and the skeleton graph features after the attention weight allocation into a preset multi-dimensional feature fusion module for multi-dimensional feature fusion to obtain human behavior features; The human behavior characteristics are classified using a preset classification function to obtain an action category probability distribution, and human action information is obtained according to the action category probability distribution.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for causing a machine to execute a method for automatic stage lighting control based on machine vision according to any one of claims 1 to 8.

10. A stage lighting automatic control system based on machine vision, characterized in that: include: a memory configured to store instructions; as well as The processor is configured to call the instruction from the memory and implement a method for automatic stage lighting control based on machine vision according to any one of claims 1 to 8 when executing the instruction.

Images