Multimedia equipment remote cooperative control method and device based on Internet of Things

Abstract

The invention discloses a multimedia equipment remote cooperative control method and device based on the Internet of Things, and relates to the technical field of remote cooperative control, and the method comprises the steps: collecting a user image of a multimedia venue, and obtaining a region triggering probability; performing user feature recognition to obtain a first probability compensation coefficient; determining a second probability compensation coefficient; compensating the regional trigger probability to obtain a regional compensation trigger probability; if the compensation triggering probability of any area exceeds a preset threshold value, a triggering area is marked, and cooperative control is carried out on light, sound effects and area special effects of the triggering area. The technical problems that in the prior art, multimedia equipment control lacks accuracy and intelligence, and a control strategy cannot be optimized in real time according to user behaviors and environment changes are solved, and accurate regional triggering probability prediction and dynamic compensation are achieved; therefore, cooperative triggering of light, sound effects and regional special effects in the multimedia venue can be controlled, and the technical effects of user experience and resource allocation are optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of remote collaborative control, and in particular to a method and device for remote collaborative control of multimedia devices based on the Internet of Things. Background Art

[0002] In modern multimedia venues, such as dome theaters, immersive experience halls, and virtual reality experience centers, the improvement of user experience increasingly depends on the precise coordinated control of multiple devices. The dynamic adjustment of lighting, sound effects, projection effects, and other multimedia elements is essential for the immersive experience. However, traditional multimedia device control systems usually trigger control effects based on fixed preset schedules or simple sensor data, often failing to take into account the variability and complexity of the scene. The main limitation of this traditional approach is that the control triggering of the device is often inflexible and cannot respond in real time to dynamic factors such as audience behavior, environmental changes, and regional traffic density. For example, user movement or changes in interest, the need for immediate adjustment of lighting and sound effects, or changes in heat in a specific area may affect the experience, but traditional control systems cannot respond instantly, resulting in an inability to effectively optimize the overall atmosphere. For example, the traffic density in a certain area may be large, but the control system fails to adjust the sound effects or lighting intensity in time, which may cause the effect to be too crowded or unbalanced, affecting the audience's immersion and experience.

[0003] The existing technology has the technical problem that multimedia device control lacks precision and intelligence, and cannot optimize control strategies in real time according to user behavior and environmental changes. Summary of the invention

[0004] The present application provides a method and apparatus for remote collaborative control of multimedia devices based on the Internet of Things, which is used to solve the technical problems in the prior art that multimedia device control lacks precision and intelligence and cannot optimize control strategies in real time according to user behavior and environmental changes.

[0005] In view of the above problems, the present application provides a method and apparatus for remote collaborative control of multimedia devices based on the Internet of Things.

[0006] The first aspect of the present application provides a remote collaborative control method for multimedia devices based on the Internet of Things, the method comprising: The user images of the multimedia venue are collected by a panoramic camera and uploaded to the central control management platform. The trigger probability of several display areas is predicted in the central control management platform according to the user image sequence to obtain several regional trigger probabilities; user feature recognition is performed based on the user image sequence, and several first probability compensation coefficients are obtained according to the feature recognition result matching; several second probability compensation coefficients are determined according to the current regional crowd density, the current time node and several regional heat ratios in the predetermined time zone; the trigger probabilities of the several areas are compensated according to the several first probability compensation coefficients and the several second probability compensation coefficients to obtain several regional compensation trigger probabilities; if any regional compensation trigger probability exceeds a predetermined threshold, the trigger area is marked, and the lighting, sound effects and regional special effects of the trigger area are collaboratively controlled through the central control management platform; wherein, several display areas are predicted in the central control management platform according to the user image sequence. The trigger probability prediction is used to obtain the trigger probabilities of several areas, including: in the central control management platform, according to the first step length and the second step length, the image frame extraction is performed on the user image sequence respectively to obtain the first initial user image set and the second user image set, wherein the second step length is 8 times of the first step length; the first initial user image set is pooled to obtain the first user image set; based on the slowfast network, a mobile feature recognition plug-in is constructed, and the first user image set and the second user image set are input into the mobile feature recognition plug-in to predict the mobile behavior, and the predicted user moving direction and the predicted user moving speed are output, wherein the mobile feature recognition plug-in is trained to convergence using the sample first image set, the sample second image set, the sample moving direction set and the sample moving speed set; according to the user's current position, the predicted user moving direction and the predicted user moving speed, the trigger probability prediction of several display areas is performed, and the trigger probabilities of several areas are output.

[0007] The second aspect of the present application provides a remote collaborative control device for multimedia devices based on the Internet of Things, the device comprising: A trigger probability acquisition module is used to collect user images of multimedia venues through panoramic cameras and upload them to the central control management platform, and predict the trigger probabilities of several display areas according to the user image sequence in the central control management platform to obtain several regional trigger probabilities; a first probability compensation coefficient acquisition module is used to perform user feature recognition based on the user image sequence, and obtain several first probability compensation coefficients according to the feature recognition result matching; a second probability compensation coefficient acquisition module is used to determine several second probability compensation coefficients according to the current regional crowd density, the current time node and several regional heat ratios in the predetermined time zone; a compensation trigger probability acquisition module is used to compensate the trigger probabilities of the several regions according to the several first probability compensation coefficients and the several second probability compensation coefficients, and obtain several regional compensation trigger probabilities; a collaborative control module is used to mark the trigger region if the compensated trigger probability of any region exceeds the predetermined threshold, and to control the trigger region through the central control management platform. The lighting, sound effects and regional special effects are collaboratively controlled; wherein the trigger probability acquisition module is also used for: in the central control management platform, according to the first step length and the second step length, respectively extracting image frames from the user image sequence to obtain a first initial user image set and a second user image set, wherein the second step length is 8 times the first step length; performing pooling processing on the first initial user image set to obtain a first user image set; constructing a mobile feature recognition plug-in based on a slowfast network, and inputting the first user image set and the second user image set into the mobile feature recognition plug-in to predict mobile behavior, and outputting a predicted user moving direction and a predicted user moving speed, wherein the mobile feature recognition plug-in is trained to convergence using a sample first image set, a sample second image set, a sample moving direction set and a sample moving speed set; predicting the trigger probabilities of several display areas according to the user's current position, the predicted user moving direction and the predicted user moving speed, and outputting several regional trigger probabilities.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages: Collect user images of multimedia venues and upload them to the central control management platform. According to the user image sequence, predict the trigger probability of several display areas and obtain several regional trigger probabilities; perform user feature recognition and obtain several first probability compensation coefficients according to the feature recognition result matching; determine several second probability compensation coefficients according to the current regional flow density, the current time node and several regional heat ratios in the predetermined time zone; compensate the trigger probabilities of the several areas according to several first probability compensation coefficients and several second probability compensation coefficients to obtain several regional compensation trigger probabilities; if any regional compensation trigger probability exceeds the predetermined threshold, mark the trigger area, and coordinately control the lighting, sound effects and regional special effects of the trigger area through the central control management platform. Accurate regional trigger probability prediction and dynamic compensation are achieved, so that the coordinated triggering of lighting, sound effects and regional special effects in multimedia venues can be controlled, and the technical effect of optimizing user experience and resource allocation can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flowchart of a method for remote collaborative control of multimedia devices based on the Internet of Things provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a remote collaborative control device for multimedia devices based on the Internet of Things provided in an embodiment of the present application.

[0011] Explanation of the reference numerals: trigger probability acquisition module 10 , first probability compensation coefficient acquisition module 20 , second probability compensation coefficient acquisition module 30 , compensation trigger probability acquisition module 40 , collaborative control module 50 . DETAILED DESCRIPTION

[0012] The present application provides a remote collaborative control method and device for multimedia devices based on the Internet of Things, which is used to solve the technical problems in the prior art that multimedia device control lacks precision and intelligence and cannot optimize control strategies in real time according to user behavior and environmental changes.

[0013] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] Embodiment 1, as Figure 1 As shown, the present application provides a remote collaborative control method for multimedia devices based on the Internet of Things, the method comprising: Step S100: collecting user images of the multimedia venue through a panoramic camera and uploading them to a central control management platform, and predicting the triggering probabilities of several display areas according to the user image sequence in the central control management platform to obtain the triggering probabilities of several areas.

[0015] Specifically, it focuses on equipment access, communication link establishment and the realization of the basic functions of the central control management platform. First, the multimedia equipment (such as projectors, audio, special effects devices, etc.), sensors (such as temperature, humidity, and light sensors for monitoring the environment), switches, and acquisition devices (mainly panoramic cameras) in the multimedia venue are connected to the Internet of Things network. Through the central control module, these access devices are identified and parameterized, each device is given a unique identifier and its initial operating parameters are set so that it can work normally in the Internet of Things environment, and a reliable communication link is established between the multimedia equipment and the central control module to ensure stable data transmission. The central control module generates various equipment control parameters based on the preset synchronization script or the real-time operation of the staff, and then accurately transmits the parameters to the target multimedia device with the help of the Internet of Things network to realize equipment operations such as special effect activation and lighting adjustment. After this series of operations are completed, the basic architecture of the central control management platform is initially built to realize the interconnection and preliminary control of the equipment. Relying on the panoramic camera installed in the multimedia venue to collect data, the panoramic camera can capture the dynamic images of users at various locations in the venue in an all-round and blind spot manner with its wide-angle shooting capability. These images will form a coherent user image sequence in chronological order. After the image sequence is obtained, it is uploaded to the central control management platform in real time through the Internet of Things. After receiving the user image sequence, the central control management platform processes the image sequence with different step lengths according to the set strategy. According to the carefully set first step length and second step length, the image frame extraction operation is performed to obtain the first initial user image set and the second user image set respectively. The second step length is deliberately set to 8 times the first step length. This difference setting is helpful to analyze user behavior characteristics from different time scales. The first user image set and the second user image set are deeply analyzed by using the mobile feature recognition plug-in built based on the slowfast network. During the analysis process, the plug-in will refer to the sample first image set, sample second image set, sample moving direction set and sample moving speed set that have been trained to convergence, so as to predict the user's moving direction and moving speed. Combined with the user's current location information in the multimedia venue, and referring to the viewing data of different user locations, moving directions and speeds in each display area in history, the long short-term memory network algorithm is used to accurately predict the triggering probability of several display areas, and finally obtain several area triggering probabilities that can reflect the possibility of the audience going to each display area. This prediction of trigger probability can allocate resources and prepare displays in advance for users' possible points of interest.

[0016] Step S200: performing user feature recognition based on the user image sequence, and obtaining a plurality of first probability compensation coefficients according to feature recognition result matching.

[0017] Specifically, a large number of sample user image sequence sets are collected first, and the group characteristics of these different samples are annotated to form a sample user feature set covering various types such as children, adults, families and couples. Using these rich sample data, the convolutional neural network (CNN) is trained. The convolutional neural network has a strong ability to extract image features. By continuously adjusting the parameters in the network, the sample user image sequence and the corresponding feature labels are repeatedly learned until the preset convergence conditions are met, and an accurate user feature recognition model is obtained. At this time, the previously acquired user image sequence is input into the trained model, and the model automatically analyzes the features such as the figure shape, clothing, action and relationship between each other in the image, and then outputs accurate feature recognition results. According to the detailed user viewing logs in the predetermined historical time zone, the correlation between the trigger probabilities of several display areas and different user features is deeply analyzed. In this process, a series of sample probability compensation coefficients are generated, and these coefficients are sorted and summarized to form several sample probability compensation coefficient sets. Based on the mapping relationship between user features and sample probability compensation coefficient sets, the first probability compensation matching library is constructed. Finally, the user features just identified are input into the matching library. According to the preset matching rules, several first probability compensation coefficients corresponding to the user features are accurately obtained from the library. These coefficients will be used for subsequent precise compensation of the regional trigger probability to improve the accuracy of the trigger probability prediction and optimize the control strategy of the multimedia venue display effect.

[0018] Step S300: Determine a plurality of second probability compensation coefficients based on the current regional flow density, the current time node, and a plurality of regional heat ratios within a predetermined time zone.

[0019] Specifically, the crowd density data of the current area is collected in real time. This data can intuitively reflect the degree of gathering of people in each area of ​​the multimedia venue. At the same time, accurate current time node information is obtained, such as the specific time, day of the week, whether it is a holiday, etc., because the habits and behavioral preferences of users may be different at different time points. According to the detailed user viewing logs in the predetermined historical time zone, the number of viewers in each display area is statistically analyzed, and the proportion of the number of viewers in each display area to the total number of viewers is calculated. These proportions are the heat ratios of several areas in the predetermined time zone, which reflects the degree of popularity of different areas by the audience in the past time period. Integrate these data, collect multiple groups of sample data including sample area crowd density set, sample time node set and sample area heat ratio set, and for these sample data, count the viewing fluctuation ratios of several display areas under the combination of different sample area crowd density, sample time node and sample area heat ratio. These fluctuation ratios are set as sample second probability compensation coefficients, thereby forming a sample second probability compensation coefficient set. The BP neural network is trained using the sample area crowd density set, sample time node set, sample area heat ratio set and sample second probability compensation coefficient set. The BP neural network continuously adjusts its own weights and thresholds to conduct deep learning on these data until the training converges. At this point, the trained BP neural network becomes the second probability compensation analyzer, and the currently collected regional flow density, time nodes, and regional heat ratios within the predetermined time zone are input into this analyzer. After complex calculations, the analyzer can output several accurate second probability compensation coefficients. These coefficients take into account the current real-time situation and historical data, providing an important basis for the subsequent adjustment of the regional trigger probability, helping to more accurately control the multimedia equipment in the display area of ​​the multimedia venue and enhance the user's viewing experience.

[0020] Step S400: compensating the plurality of regional trigger probabilities according to the plurality of first probability compensation coefficients and the plurality of second probability compensation coefficients to obtain a plurality of regional compensated trigger probabilities.

[0021] Specifically, when using the decision tree algorithm to obtain the regional compensation trigger probability, a large amount of historical data must first be collected. These data include the regional trigger probability, the first probability compensation coefficient, the second probability compensation coefficient, and the corresponding actual or simulated regional compensation trigger probability in different scenarios. Then, the regional trigger probability, the first probability compensation coefficient, and the second probability compensation coefficient are used as input features of the decision tree, and these historical data are used as training sets to build a decision tree model. During the training process, the decision tree will determine which feature to select for splitting at each node by calculating the information gain or information gain rate index, thereby forming a tree structure and learning the complex relationship between these features and the regional compensation trigger probability. After the model training is completed, the new regional trigger probability, the first probability compensation coefficient, and the second probability compensation coefficient data are input into the trained decision tree model. The decision tree will judge the input data from the root node according to the constructed tree structure and splitting rules, and go down along the corresponding branches. Finally, the predicted regional compensation trigger probability is obtained at the leaf node, providing an accurate basis for the subsequent control of the multimedia venue display area.

[0022] Step S500: If the compensation trigger probability of any area exceeds a predetermined threshold, the trigger area is marked, and the lighting, sound effects and regional special effects of the trigger area are collaboratively controlled through the central control management platform.

[0023] Specifically, the compensation trigger probability of each area is continuously monitored, and the compensation trigger probability of each area is compared with the pre-set threshold one by one. Once it is found that the compensation trigger probability of a certain area is greater than this predetermined threshold, the area will be marked as a trigger area. After determining the trigger area, the central control management platform quickly starts the collaborative control program, and will quickly obtain the average control delay of the lighting, sound effects and regional special effects. The average control delay refers to the average time required from sending a control instruction to the corresponding device performing an action. Based on these average control delays, the collaborative control nodes are accurately set to ensure that various control operations can be accurately synchronized. Subsequently, according to the pre-established control plan, at the set collaborative control node, the central control management platform sends a control signal to the lighting equipment, audio equipment and regional special effects equipment in the trigger area at the collaborative control node, and the lighting, sound effects and regional special effects in the trigger area are collaboratively controlled. In terms of lighting control, the lighting intensity is adjusted according to preset requirements. The lights may be brightened to highlight the display area, or dimmed to create a specific atmosphere. At the same time, the lighting focus area is precisely controlled to guide the audience's line of sight. In terms of sound effect control, the sound effect intensity is adjusted to make the sound volume adapt to the display scene, and the propagation direction of the sound effect is controlled. For regional special effects, at least special effects including smoke and wind effects are activated. For example, smoke is created in the trigger area of ​​the simulated starry sky scene to simulate the nebula effect. Through multi-faceted coordination and cooperation, the user experience is improved.

[0024] In a possible implementation, step S100 further includes: Step S110: In the central control management platform, image frames are extracted from the user image sequence according to the first step length and the second step length respectively to obtain a first initial user image set and a second user image set, wherein the second step length is 8 times the first step length.

[0025] Step S120: performing pooling processing on the first initial user image set to obtain a first user image set.

[0026] Step S130: construct a mobile feature recognition plug-in based on the slowfast network, and input the first user image set and the second user image set into the mobile feature recognition plug-in to predict mobile behavior, and output a predicted user movement direction and a predicted user movement speed, wherein the mobile feature recognition plug-in is trained to convergence using a sample first image set, a sample second image set, a sample movement direction set, and a sample movement speed set.

[0027] Step S140: predicting the triggering probabilities of several display areas according to the user's current location, the predicted user moving direction and the predicted user moving speed, and outputting several area triggering probabilities.

[0028] Specifically, after receiving the user image sequence from the panoramic camera, the central control management platform extracts the image frames of the user image sequence collected by the panoramic camera according to the predetermined first step length and second step length. The setting of the first step length and the second step length is carefully considered, and the second step length is deliberately set to 8 times the first step length. This difference setting helps to obtain image information from different time scales. In this way, the first initial user image set and the second user image set are obtained respectively. The first initial user image set can reflect the details of the user's behavior in a shorter time interval, while the second user image set can show the user's behavior change trend from a more macroscopic time perspective.

[0029] The first initial user image set is pooled. The pooling operation can reduce the dimension of the data without losing key information, reduce the amount of subsequent calculations, and enhance the robustness of the model to image translation, rotation and other transformations. After the pooling process, the first user image set is successfully obtained, providing more efficient and representative data for subsequent analysis.

[0030] A mobile feature recognition plug-in was built based on the slowfast network. The slowfast network has a unique structural design and can process information of different time scales at the same time, which is very suitable for analyzing the user's mobile behavior in multimedia venues. The mobile feature recognition plug-in is trained using the sample first image set, sample second image set, sample moving direction set, and sample moving speed set. During the training process, the parameters of the plug-in are continuously adjusted so that it can accurately learn the association between image data and user movement behavior until the plug-in is trained to a convergent state. When the plug-in training is completed, the first user image set and the second user image set obtained previously are input into the mobile feature recognition plug-in. The plug-in will conduct an in-depth analysis of these image data and output the predicted user moving direction and predicted user moving speed, providing key information for subsequent trigger probability prediction.

[0031] The user's current location, predicted user movement direction and predicted user movement speed are used as important bases to predict the triggering probability of several display areas. The user's current location, possible movement direction and speed are comprehensively considered, and the long short-term memory network is used to analyze the possibility of the user reaching each display area in the future, and finally output several area triggering probabilities. These triggering probabilities will provide strong data support for the subsequent display effect control and resource allocation of multimedia venues, enabling theaters to make precise adjustments and optimizations based on user behavior trends.

[0032] In a possible implementation, step S140 further includes: Step S141: According to the user viewing log of the multimedia venue, a sample user position set, a sample user moving direction set and a sample user moving speed set are collected, and the viewing proportions of different sample user positions, different sample user moving directions and different sample user moving speeds in several display areas are counted and set as several sample trigger probabilities to obtain several sample trigger probability sets.

[0033] Step S142: using the sample user position set, the sample user moving direction set, the sample user moving speed set and a plurality of sample trigger probability sets to train a long short-term memory network until convergence, and constructing a regional trigger probability predictor.

[0034] Step S143: input the user's current location, the predicted user moving direction and the predicted user moving speed into the area trigger probability predictor, and output the plurality of area trigger probabilities.

[0035] Specifically, we deeply explore the user viewing logs of multimedia venues, which record in detail the activity trajectories and viewing behaviors of the audience in the theater. Through detailed analysis of the logs, we collected a set of sample user locations, covering the specific location information of different audiences at different times in the theater; a set of sample user movement directions, reflecting the direction of movement of the audience at different time points; and a set of sample user movement speeds, reflecting the speed of the audience's movement. We deeply process these collected sample data, and count the viewing ratios of different sample user locations, different sample user movement directions, and different sample user movement speeds in several display areas. These viewing ratios are defined as sample trigger probabilities, and thus several sample trigger probability sets are obtained. These probability sets can intuitively reflect the degree of attention paid by the audience to each display area in different states in the past.

[0036] The regional trigger probability predictor is constructed. The sample data collected and sorted are used to import the sample user location set, sample user movement direction set, sample user movement speed set and several sample trigger probability sets into the training environment of the long short-term memory network (LSTM). The long short-term memory network has a unique gating structure, including input gate, forget gate and output gate, which enables it to effectively handle the long-term dependency problem in time series data and is very suitable for analyzing the complex relationship between the user movement behavior and the regional trigger probability in multimedia venues. During the training process, the network takes the sample user location, movement direction and movement speed as input features and the sample trigger probability as the target output. Based on these input data, the network will minimize the error between the predicted value and the actual sample trigger probability by continuously adjusting its own weights and bias parameters. In each iterative training, the network will fine-tune the parameters to improve the recognition ability of data patterns and the prediction accuracy. As the training continues, the error will gradually decrease. When the error reaches the preset minimum value or the error changes very little in multiple consecutive rounds of training, it indicates that the network has learned the inherent laws in the data and the training has reached a convergence state. At this point, the regional trigger probability predictor is successfully constructed. The predictor can accurately predict the probability of viewers going to each display area based on the input user-related information.

[0037] The current actual data, i.e. the user's current location, the user's moving direction and moving speed predicted in the previous steps, are input into the constructed area trigger probability predictor, which quickly and accurately analyzes and calculates the input data, and finally outputs several area trigger probabilities. These trigger probabilities show the probability of the user going to each display area in the current state.

[0038] In a possible implementation, step S200 further includes: Step S210: Collect a set of sample user image sequences, and perform group feature annotation on different sample user image sequences to obtain a sample user feature set, wherein user features at least include children, adults, families, and couples.

[0039] Step S220: using the sample user image sequence set and the sample user feature set to train a convolutional neural network, obtaining a user feature recognition model that meets the convergence condition, performing user feature recognition on the user image sequence, and outputting a feature recognition result.

[0040] Specifically, we collect images of user behaviors in different time periods and scenarios in an all-round way, so as to obtain a large number of sample user image sequence sets, which cover a rich variety of user groups and scene information. In order to enable the subsequent model training to effectively identify the characteristics of different user groups, group feature annotation is performed on each sample user image sequence. During the annotation process, user features are divided into at least categories such as children, adults, families, and couples, ensuring that each sample user image sequence is accurately labeled with the corresponding user feature information. After this operation, the sample user feature set is successfully obtained, providing high-quality training data with clear labels for subsequent model training.

[0041] The sample user image sequence set is used as input data, and the sample user feature set is used as the corresponding label data. They are input into the convolutional neural network for training. The convolutional neural network has a powerful image feature extraction capability. Through the combination of multiple convolutional layers, pooling layers and fully connected layers, it can automatically learn the complex feature patterns in the image. During the training process, the network will continuously adjust its weights and bias parameters, try to perform feature recognition on the input sample user image sequence, and compare the recognition results with the corresponding sample user feature set to calculate the error between the two. Then, the back propagation algorithm is used to adjust the network parameters according to the size of the error, so that the error gradually decreases. After multiple rounds of training iterations, when the network error reaches the preset convergence condition, it indicates that the network has fully learned the rules and features in the sample data. At this time, a user feature recognition model that meets the convergence conditions is trained. The collected real-time user image sequence is input into this trained model. The model will analyze and recognize the input image based on the feature pattern it has learned, and finally output accurate feature recognition results, providing strong user feature information support for subsequent decision-making.

[0042] In a possible implementation, step S200 further includes: Step S230: According to the user viewing logs in the predetermined historical time zone, trigger probability correlation analysis is performed on a plurality of display areas and user characteristics respectively, and a plurality of sample probability compensation coefficient sets are output.

[0043] Step S240: construct a first probability compensation matching library based on the mapping relationship between the user features and a plurality of sample probability compensation coefficient sets, and input the feature recognition result into the first probability compensation matching library to obtain a plurality of first probability compensation coefficients.

[0044] Specifically, the user viewing logs in the predetermined historical time zone are extracted. These logs record the activity trajectory, viewing behavior and other information of each user in detail. These log data are preprocessed and organized into a structured format that is easy to analyze. For each display area, the key data such as the number of views and length of stay of users with different user characteristics (such as children, adults, families, couples, etc.) in the area are counted separately. Then, based on these statistical data, statistical methods and conditional probability calculation are used to analyze the triggering probability of users with different user characteristics in each display area. For example, to calculate the triggering probability of child users in display area A, the number of views of child users in display area A is divided by the total number of views of child users in all display areas. In this way, the degree of correlation between different user characteristics and the triggering probability of each display area is calculated, and then several sets of sample probability compensation coefficients are obtained. These coefficient sets reflect the influence of different user characteristics on the triggering probability of each display area, providing key data support for subsequent precise control.

[0045] Build the first probability compensation matching library, create a data structure specifically for storing mapping relationships, such as using a relational database (such as MySQL), create a new table in the database, and the fields of the table include user feature fields and several sample probability compensation coefficient fields corresponding to different display areas. Next, insert the mapping relationship between the obtained user features and the sample probability compensation coefficient set into this table row by row. For example, if there are several user features such as children, adults, families, and couples, for each feature, there is a corresponding probability compensation coefficient in each display area, and fill this information into the table one by one. If Python is used for development, it is implemented using a dictionary data structure, with user features as the key of the dictionary, and a list of corresponding sample probability compensation coefficients as the value to build a nested dictionary. After completing the construction of the first probability compensation matching library, use the feature recognition result as the retrieval basis. If it is a matching library built based on a database, execute an SQL query statement, use the feature recognition result as the query condition, search for matching records in the table, and then extract several first probability compensation coefficients from the record; if the Python dictionary method is used, directly use the feature recognition result as the key to obtain the corresponding value from the dictionary, that is, to obtain several first probability compensation coefficients. Through this specific implementation method, the matching process from feature recognition results to the first probability compensation coefficient can be completed quickly and accurately, providing strong support for the subsequent precise adjustment of the regional trigger probability.

[0046] In a possible implementation, step S300 further includes: Step S310: according to the user viewing logs in the predetermined historical time zone, the proportion of the number of viewers in several display areas in the predetermined historical time zone is counted respectively, and set as several regional heat ratios.

[0047] Step S320: Collect a set of sample area crowd density, a set of sample time nodes, and a set of sample area heat ratios, and count the viewing fluctuation ratios of several display areas under different sample area crowd density, sample time nodes, and sample area heat ratios, set them as several sample second probability compensation coefficients, and obtain a set of sample second probability compensation coefficients.

[0048] Step S330: Train the BP neural network using the sample area crowd density set, sample time node set, sample area heat ratio set and sample second probability compensation coefficient set until convergence to obtain a second probability compensation analyzer, and obtain several second probability compensation coefficients based on the current area crowd density, current time node and several area heat ratio predictions.

[0049] Specifically, we conduct in-depth mining of user viewing logs in a predetermined historical time zone, comprehensively sort out the viewing data of each display area recorded in the log, count the number of viewers in each display area during the time period, and compare the number of viewers in each display area with the total number of viewers, so as to obtain the proportion of viewers in each display area, which is defined as several regional popularity ratios. Through this process, we can clearly understand the popularity of each display area in the past time period.

[0050] The data collection work is fully launched. Through the sensor equipment installed in each display area of ​​the multimedia venue, the sample area flow density set within a period of time is collected in real time. These data can accurately reflect the density of people in different display areas at different times; at the same time, the sample time node set is collected, which contains the accurate time information of each time period; and the sample area heat ratio set that has been statistically obtained is extracted, which reflects the popularity of each display area in the predetermined historical time zone. After collecting these data, all possible data combinations are traversed and analyzed. For each different combination of sample area flow density, sample time node and sample area heat ratio, the viewing fluctuation ratio of several display areas under this combination is deeply counted. The viewing fluctuation ratio is calculated by comparing the changes in the number of viewers in each display area in different time periods. It reflects the increase or decrease of users' viewing interest in each display area under specific conditions of population density, time and regional heat. The viewing fluctuation ratios obtained from these statistics are set as several sample second probability compensation coefficients, and then all the sample second probability compensation coefficients are sorted and summarized to form a sample second probability compensation coefficient set. These data will provide key data support for the subsequent construction of the second probability compensation analyzer, so as to more accurately optimize the display effect of the multimedia venue.

[0051] The BP neural network is selected as the training model. The BP neural network has a strong nonlinear mapping ability and can effectively handle complex data analysis tasks. The sample area flow density set, sample time node set, and sample area heat ratio set are used as input data, and the sample second probability compensation coefficient set is used as output data, which are input into the BP neural network for training. During the training process, the BP neural network minimizes the error between the predicted value and the actual sample second probability compensation coefficient by continuously adjusting its own weights and thresholds. After multiple rounds of training iterations, when the error reaches the preset convergence condition, it indicates that the network has fully learned the intrinsic relationship between the input data and the output data. At this time, the second probability compensation analyzer is successfully constructed. The current area flow density, current time node, and several area heat ratios are input into this analyzer. The analyzer uses the rules and patterns learned during the training process to analyze and calculate the input data, and finally predicts several second probability compensation coefficients, which provide an important basis for the subsequent display area trigger probability compensation.

[0052] In a possible implementation, step S500 further includes: Step S510: setting the display area where the area compensation trigger probability exceeds a predetermined threshold as a trigger area.

[0053] Step S520: Obtain the average control delay of lighting, sound effects, and regional special effects, and set a collaborative control node according to the average control delay.

[0054] Step S530: According to a predetermined control scheme, the lighting equipment, audio equipment and regional special effect equipment of the trigger area are collaboratively controlled by the central control management platform under the collaborative control node, wherein the lighting control includes the light intensity and the light focus area, the sound effect control includes the sound effect intensity and the propagation direction, and the regional special effects include at least smoke and wind effects.

[0055] Specifically, the regional compensation trigger probability of each display area is continuously monitored in real time, which comprehensively reflects the possibility of the audience going to this area. At the same time, a reasonable predetermined threshold is set in advance, which is determined based on multiple factors such as the operating experience of the multimedia venue, the audience's behavioral habits, and the display effect requirements. During the monitoring process, the regional compensation trigger probability of each display area is compared with the predetermined threshold one by one. Once it is found that the regional compensation trigger probability of a certain display area is greater than this predetermined threshold, the display area will be quickly marked as a trigger area. This mark means that the display area has a high probability of attracting the audience's attention, and a series of display effect optimization and equipment collaborative control operations will need to be performed on it in the future to better meet user needs.

[0056] Start collecting and analyzing the average control delay data of lighting, sound effects, and regional special effects equipment. For lighting equipment, in the past period of time, control instructions such as adjusting the light intensity and focusing area were randomly sent to the lighting control system many times. At the same time, the precise time point when each instruction was issued and the time point when the lighting equipment actually responded and completed the corresponding action were recorded. By calculating the difference between these two time points, the delay data of each control operation was obtained. After a large number of test records, these delay data were summarized and their average values ​​were calculated using mathematical statistical methods to obtain the average control delay of the lighting equipment. For sound effect equipment, a similar method was used to send control instructions such as adjusting the volume and changing the propagation direction to the sound system many times. The time when the instruction was issued and the actual response time of the sound effect equipment were also accurately recorded. The delay of each operation was calculated, and finally the average control delay of the sound effect equipment was calculated. This delay data reflects the average time required from the issuance of the instruction to the completion of the sound system to adjust the volume, change the direction, etc. In terms of regional special effects, for special effects equipment such as smoke and wind effects, control instructions for triggering special effects are sent multiple times, the time when the instructions are issued and the time when the special effects are actually started are recorded, and the average value of these time differences is calculated to obtain the average control delay of the regional special effects equipment. After obtaining the average control delay data of lighting, sound effects and regional special effects, the setting of collaborative control nodes is carried out. Taking these average control delay data as the key basis, precise calculations and planning are carried out to sort out the display process of multimedia venues, and clarify the roles that lighting, sound effects and regional special effects need to play in different display links and the order of coordination between them. For example, in the opening display link, it may be expected that the lights will first focus on the center of the stage to attract the audience's attention, and then the sound effects will gradually increase to create an atmosphere, and finally the regional special effects will be started to enhance the visual impact. Based on this coordinated order, according to the average control delay of each device, their start time points are determined. Assuming that the average control delay of lighting is 0.5 seconds, the average control delay of sound effects is 1 second, and the average control delay of regional special effects is 1.5 seconds, in order to achieve a smooth opening effect, the lighting is set to receive control instructions 0.5 seconds before the start of the show, the sound effects receive instructions 1 second before the start of the show, and the regional special effects receive instructions 1.5 seconds before the start of the show. In this way, the start time points of different devices, that is, the collaborative control nodes, can be accurately set to ensure that during the show, each device can be accurately started and work together in accordance with the predetermined order and time.

[0057] When the collaborative control node set according to the average control delay is reached, the central control management platform immediately carries out collaborative control operations for the lighting equipment, audio equipment and regional special effect equipment in the triggering area according to the pre-established control plan. In terms of lighting equipment control, the lighting intensity and focus area requirements in the predetermined control plan will be precisely adjusted. If the plan requires highlighting when displaying important pictures, the lighting intensity will be increased to make the triggering area brighter and more eye-catching; at the same time, the light will be accurately focused on the predetermined display area to guide the user's sight to focus on it, thereby enhancing the visual effect. In terms of audio equipment control, the intensity and propagation direction of the sound effect are adjusted according to the planning of the predetermined plan. For example, when performing fierce battle scenes, the intensity of the sound effect is increased to create a shocking auditory atmosphere; and according to the location and distribution of the user, the propagation direction of the sound effect is accurately controlled so that the user can feel a more three-dimensional and realistic sound effect. As for the regional special effect equipment, according to the predetermined plan, special effects including at least smoke and wind effects are started at the collaborative control node. When simulating natural scenes, smoke is released at the right time to create a misty effect; at the same time, wind effect equipment is started to further enhance the audience's interactive experience.

[0058] Embodiment 2 is based on the same inventive concept as the method for remote collaborative control of multimedia devices based on the Internet of Things in the aforementioned embodiment. Figure 2 As shown, the present application provides a remote collaborative control device for multimedia devices based on the Internet of Things, and the device and method embodiments in the embodiments of the present application are based on the same inventive concept. Among them, the device includes: The trigger probability acquisition module 10 is used to collect user images of the multimedia venue through a panoramic camera and upload them to the central control management platform, and predict the trigger probability of several display areas according to the user image sequence in the central control management platform to obtain the trigger probabilities of several areas.

[0059] The first probability compensation coefficient acquisition module 20 is used to perform user feature recognition based on the user image sequence, and obtain a plurality of first probability compensation coefficients according to the feature recognition result matching.

[0060] The second probability compensation coefficient acquisition module 30 is used to determine a plurality of second probability compensation coefficients according to the current regional flow density, the current time node and a plurality of regional heat ratios within a predetermined time zone.

[0061] The compensation trigger probability acquisition module 40 is used to compensate the plurality of regional trigger probabilities according to the plurality of first probability compensation coefficients and the plurality of second probability compensation coefficients to acquire a plurality of regional compensation trigger probabilities.

[0062] The collaborative control module 50 is used to mark the trigger area if the compensation trigger probability of any area exceeds a predetermined threshold, and to collaboratively control the lighting, sound effects and regional special effects of the trigger area through the central control management platform.

[0063] Wherein, the trigger probability acquisition module 10 is also used for: In the central control management platform, image frames of the user image sequence are extracted according to the first step length and the second step length respectively to obtain the first initial user image set and the second user image set, wherein the second step length is 8 times the first step length. The first initial user image set is pooled to obtain the first user image set. A mobile feature recognition plug-in is constructed based on the slowfast network, and the first user image set and the second user image set are input into the mobile feature recognition plug-in to predict mobile behavior, and the predicted user moving direction and the predicted user moving speed are output, wherein the mobile feature recognition plug-in is trained to convergence using the sample first image set, the sample second image set, the sample moving direction set and the sample moving speed set. The trigger probability of several display areas is predicted according to the user's current position, the predicted user moving direction and the predicted user moving speed, and several area trigger probabilities are output.

[0064] Furthermore, the area trigger probability output unit also includes: The sample trigger probability set acquisition unit is used to collect a sample user position set, a sample user moving direction set and a sample user moving speed set according to the user viewing log of the multimedia venue, and to count the viewing proportions of different sample user positions, different sample user moving directions and different sample user moving speeds in several display areas, set them as several sample trigger probabilities, and obtain several sample trigger probability sets.

[0065] The regional trigger probability predictor construction unit is used to train the long short-term memory network using the sample user position set, the sample user moving direction set, the sample user moving speed set and several sample trigger probability sets until convergence, so as to construct the regional trigger probability predictor.

[0066] The area trigger probability predictor output unit is used to input the user's current position, the predicted user movement direction and the predicted user movement speed into the area trigger probability predictor, and output the plurality of area trigger probabilities.

[0067] Furthermore, the first probability compensation coefficient acquisition module 20 further includes: The sample user feature set acquisition unit is used to collect a sample user image sequence set and perform group feature annotation on different sample user image sequences to obtain a sample user feature set, wherein user features at least include children, adults, families and couples.

[0068] The feature recognition result output unit is used to train a convolutional neural network using the sample user image sequence set and the sample user feature set to obtain a user feature recognition model that meets the convergence conditions, perform user feature recognition on the user image sequence, and output a feature recognition result.

[0069] Furthermore, the first probability compensation coefficient acquisition module 20 further includes: The sample probability compensation coefficient set output unit is used to perform trigger probability correlation analysis on a number of display areas and user features according to user viewing logs in a predetermined historical time zone, and output a number of sample probability compensation coefficient sets.

[0070] The first probability compensation matching library construction unit is used to construct a first probability compensation matching library based on the mapping relationship between the user characteristics and a plurality of sample probability compensation coefficient sets, and input the characteristic recognition result into the first probability compensation matching library to obtain a plurality of first probability compensation coefficients.

[0071] Furthermore, the second probability compensation coefficient acquisition module 30 further includes: The regional heat ratio setting unit is used to count the proportion of the number of viewers in several display areas in the predetermined historical time zone according to the user viewing logs in the predetermined historical time zone, and set them as several regional heat ratios.

[0072] The sample second probability compensation coefficient set acquisition unit is used to collect the sample area crowd density set, the sample time node set and the sample area heat ratio set, and count the viewing fluctuation ratios of several display areas under different sample area crowd density, sample time node and sample area heat ratio, set them as several sample second probability compensation coefficients, and obtain the sample second probability compensation coefficient set.

[0073] The second probability compensation analyzer acquisition unit is used to train the BP neural network using the sample area crowd density set, the sample time node set, the sample area heat ratio set and the sample second probability compensation coefficient set until convergence, to obtain the second probability compensation analyzer, and to obtain several second probability compensation coefficients according to the current area crowd density, the current time node and several area heat ratio predictions.

[0074] Furthermore, the collaborative control module 50 also includes: The trigger area setting unit is used to set the display area where the area compensation trigger probability exceeds a predetermined threshold as the trigger area.

[0075] The collaborative control node setting unit is used to obtain the average control delay of lighting, sound effects and regional special effects, and set the collaborative control node according to the average control delay.

[0076] A collaborative control unit is used to collaboratively control the lighting equipment, audio equipment and regional special effect equipment of the trigger area under the collaborative control node through the central control management platform in accordance with a predetermined control scheme, wherein the lighting control includes the lighting intensity and the lighting focus area, the sound effect control includes the sound effect intensity and the propagation direction, and the regional special effects include at least smoke and wind effects.

[0077] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0078] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0079] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A remote collaborative control method for multimedia devices based on the Internet of Things, characterized in that: Methods include: The user images of the multimedia venue are collected by a panoramic camera and uploaded to the central control management platform, and the trigger probability of several display areas is predicted in the central control management platform according to the user image sequence to obtain the trigger probabilities of several areas; Performing user feature recognition based on the user image sequence, and obtaining a plurality of first probability compensation coefficients according to feature recognition result matching; Determine a plurality of second probability compensation coefficients according to the current regional flow density, the current time node, and the heat ratio of a plurality of regions within a predetermined time zone; Compensating the plurality of regional trigger probabilities according to the plurality of first probability compensation coefficients and the plurality of second probability compensation coefficients to obtain a plurality of regional compensation trigger probabilities; If the probability of triggering compensation in any area exceeds a predetermined threshold, the triggering area is marked, and the lighting, sound effects and regional special effects of the triggering area are collaboratively controlled through the central control management platform; Among them, the trigger probability prediction of several display areas is performed according to the user image sequence in the central control management platform to obtain several area trigger probabilities, including: In the central control management platform, image frames are extracted from the user image sequence according to the first step length and the second step length respectively to obtain a first initial user image set and a second user image set, wherein the second step length is 8 times the first step length; Performing pooling processing on the first initial user image set to obtain a first user image set; A mobile feature recognition plug-in is constructed based on a slowfast network, and the first user image set and the second user image set are input into the mobile feature recognition plug-in to predict mobile behavior, and a predicted user moving direction and a predicted user moving speed are output, wherein the mobile feature recognition plug-in is trained to convergence using a sample first image set, a sample second image set, a sample moving direction set, and a sample moving speed set; The trigger probability of several display areas is predicted according to the user's current position, the predicted user movement direction and the predicted user movement speed, and several area trigger probabilities are output.

2. The method for remote collaborative control of multimedia devices based on the Internet of Things according to claim 1, characterized in that: The trigger probability prediction of several display areas is performed according to the user's current location, the predicted user moving direction and the predicted user moving speed, including: According to the user viewing logs of the multimedia venue, a sample user location set, a sample user moving direction set and a sample user moving speed set are collected, and the viewing proportions of different sample user locations, different sample user moving directions and different sample user moving speeds in several display areas are counted and set as several sample trigger probabilities to obtain several sample trigger probability sets; The sample user position set, the sample user moving direction set, the sample user moving speed set and a plurality of sample trigger probability sets are used to train a long short-term memory network until convergence, thereby constructing a regional trigger probability predictor; The user's current position, the predicted user moving direction and the predicted user moving speed are input into the area trigger probability predictor, and the plurality of area trigger probabilities are output.

3. The method for remote collaborative control of multimedia devices based on the Internet of Things according to claim 1, characterized in that: Performing user feature recognition based on the user image sequence includes: Collecting a sample user image sequence set, and annotating different sample user image sequences with group characteristics to obtain a sample user feature set, wherein the user characteristics at least include children, adults, families, and couples; The sample user image sequence set and the sample user feature set are used to train a convolutional neural network to obtain a user feature recognition model that meets the convergence condition, perform user feature recognition on the user image sequence, and output a feature recognition result.

4. The method for remote collaborative control of multimedia devices based on the Internet of Things according to claim 3 is characterized in that: According to the feature recognition result matching, a plurality of first probability compensation coefficients are obtained, including: According to the user viewing logs in the predetermined historical time zone, trigger probability correlation analysis is performed on several display areas and user characteristics, and several sample probability compensation coefficient sets are output; A first probability compensation matching library is constructed based on the mapping relationship between the user features and a plurality of sample probability compensation coefficient sets, and the feature recognition result is input into the first probability compensation matching library to obtain a plurality of first probability compensation coefficients through matching.

5. The method for remote collaborative control of multimedia devices based on the Internet of Things according to claim 1, characterized in that: According to the current regional flow density, the current time node and the heat ratio of several regions in the predetermined time zone, several second probability compensation coefficients are determined, including: According to the user viewing logs in the predetermined historical time zone, the proportion of the number of viewers in several display areas in the predetermined historical time zone is counted respectively, and set as several regional heat ratios; Collect a set of sample area crowd density, a set of sample time nodes, and a set of sample area heat ratios, and count the viewing fluctuation ratios of several display areas under different sample area crowd density, sample time nodes, and sample area heat ratios, set them as several sample second probability compensation coefficients, and obtain a set of sample second probability compensation coefficients; The BP neural network is trained using the sample area crowd density set, sample time node set, sample area heat ratio set and sample second probability compensation coefficient set until convergence to obtain a second probability compensation analyzer, and several second probability compensation coefficients are predicted based on the current area crowd density, current time node and several area heat ratios.

6. The method for remote collaborative control of multimedia devices based on the Internet of Things according to claim 1, characterized in that: If the probability of triggering compensation in any area exceeds a predetermined threshold, the triggering area is marked, and the lighting, sound effects, and area special effects of the triggering area are collaboratively controlled through the central control management platform, including: The display area whose regional compensation trigger probability exceeds a predetermined threshold is set as a trigger area; Obtain the average control delay of lighting, sound effects, and regional special effects, and set the collaborative control node according to the average control delay; According to a predetermined control scheme, the lighting equipment, audio equipment and regional special effect equipment of the trigger area are collaboratively controlled under the collaborative control node through the central control management platform, wherein the lighting control includes the light intensity and the light focus area, the sound effect control includes the sound effect intensity and the propagation direction, and the regional special effects include at least smoke and wind effects.

7. A remote collaborative control device for multimedia devices based on the Internet of Things, characterized in that: The device is used to implement the method for remote collaborative control of multimedia devices based on the Internet of Things according to any one of claims 1 to 6, and the device includes: A trigger probability acquisition module is used to collect user images of the multimedia venue through a panoramic camera and upload them to the central control management platform, and predict the trigger probabilities of several display areas according to the user image sequence in the central control management platform to obtain the trigger probabilities of several areas; A first probability compensation coefficient acquisition module, used to perform user feature recognition based on the user image sequence, and obtain a plurality of first probability compensation coefficients according to feature recognition result matching; A second probability compensation coefficient acquisition module is used to determine a plurality of second probability compensation coefficients according to the current regional flow density, the current time node, and the heat ratio of a plurality of regions in a predetermined time zone; A compensation trigger probability acquisition module, used to compensate the plurality of regional trigger probabilities according to the plurality of first probability compensation coefficients and the plurality of second probability compensation coefficients, and acquire a plurality of regional compensation trigger probabilities; A collaborative control module, for marking a triggering area if the probability of compensation triggering in any area exceeds a predetermined threshold, and collaboratively controlling the lighting, sound effects and area special effects of the triggering area through the central control management platform; Wherein, the trigger probability acquisition module is also used for: In the central control management platform, image frames are extracted from the user image sequence according to the first step length and the second step length respectively to obtain a first initial user image set and a second user image set, wherein the second step length is 8 times the first step length; Performing pooling processing on the first initial user image set to obtain a first user image set; A mobile feature recognition plug-in is constructed based on a slowfast network, and the first user image set and the second user image set are input into the mobile feature recognition plug-in to predict mobile behavior, and a predicted user moving direction and a predicted user moving speed are output, wherein the mobile feature recognition plug-in is trained to convergence using a sample first image set, a sample second image set, a sample moving direction set, and a sample moving speed set; The trigger probability of several display areas is predicted according to the user's current position, the predicted user movement direction and the predicted user movement speed, and several area trigger probabilities are output.

Images