Large-screen touch interaction conference room multimedia terminal based on machine vision

Abstract

The invention discloses a large-screen touch interaction conference room multimedia terminal based on machine vision, which relates to the technical field of multimedia terminal image processing and comprises an acquisition module, an extraction module, a processing module and an identification interaction module. According to the method, the characteristic value of the target object in the large-screen display content at the specific moment or position can be accurately calculated based on the machine vision and the excellent algorithm, and multiple factors such as the width, the height, the color value and the total pixel number of the target object can be synthesized, so that accurate extraction of the characteristics of the target object is realized; therefore, the gesture recognition precision is remarkably improved, the gesture action of the user can be intelligently recognized, and high recognition accuracy can be kept in a complex scene, so that the interaction intention of the user can be more accurately understood, the intelligence of the system and the user experience are improved, and the user experience is improved. According to the method, various characteristic values and gesture recognition results can be synthesized, so that intelligent processing of a final interaction decision is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia terminal image processing, in particular to a large-screen touch interactive conference room multimedia terminal based on machine vision. Background Art

[0002] With the rapid development of information technology, conference room multimedia terminals have become an indispensable part of the modern office environment. Especially in large enterprises and institutions, conference room multimedia terminals not only undertake the functions of display and demonstration, but also gradually incorporate more interactive elements to improve meeting efficiency and user experience. Among them, conference room multimedia terminals based on machine vision large-screen touch interaction have received widespread attention and application due to their intuitive and convenient operation methods.

[0003] The multimedia terminal is equipped with a large-size high-definition display screen that can clearly display various conference materials, videos, charts and other content. At the same time, through machine vision technology, the system can capture and analyze user gestures in real time, thereby realizing touch interactive operations on multimedia content. This interactive method not only improves the interactivity of the meeting, but also enables users to control the meeting process more flexibly, thereby improving meeting efficiency.

[0004] However, currently available multimedia terminals may find it difficult to accurately extract the feature values ​​of target objects in large-screen display content, which in turn limits the accuracy of gesture recognition. In addition, due to the lack of effective algorithm support, existing multimedia terminal systems may find it difficult to accurately identify user gestures, especially in complex scenarios. In addition, existing terminals may lack sufficient intelligence when interacting, making it difficult to make reasonable decisions based on actual user needs. Summary of the invention

[0005] The purpose of the present invention is to provide a large-screen touch interactive conference room multimedia terminal based on machine vision, which solves the problems raised in the above-mentioned background technology.

[0006] To achieve the above purpose, the present invention provides the following technical solution: a large-screen touch interactive conference room multimedia terminal based on machine vision, comprising:

[0007] Acquisition module: captures dynamic images in the large-screen touch interaction area and the contact status information between fingers and the screen;

[0008] Extraction module: used to input the dynamic image and the contact state information between the finger and the screen in the large-screen touch interaction area, and output the width value of the target object at time t and position p, the height value of the target object at time t and position p, and the distance from the target object to the screen at time t and position p;

[0009] Processing module: used for inputting the width value of the target object at time t and position p, the height value of the target object at time t and position p, and the distance from the target object to the screen at time t and position p, and the processing module performs data cleaning and sorting on the input parameters;

[0010] Identification and interaction module: used to input the width value of the target object at time t and position p, the height value of the target object at time t and position p, and the distance from the target object to the screen at time t and position p after being processed by the processing module, output the feature value of the target object at time t and position p, the interaction intention with the gesture recognition area q at time t, and the final interaction decision at time t, and upload the output results of the interaction module to the management terminal, which performs corresponding operations and responses according to the final interaction decision at time t.

[0011] Optionally, the identification interaction module includes: a feature submodule, an interaction submodule, a comprehensive submodule and an iteration submodule.

[0012] Optionally, the calculation formula of the characteristic submodule is as follows:

[0013]

[0014] in:

[0015] FPOD t，p Refers to the feature value of the target object at time t and position p;

[0016] A1 t，p Refers to the width of the target object at time t and position p, A2 t，p Refers to the height value of the target object at time t and position p, NOC t，p Refers to the total number of pixels in the target object at time t and position p, YAS t，p，i Refers to the red color value of pixel i of the target object at time t and position p, YBS t，p，i Refers to the green color value of pixel i of the target object at time t and position p, YCS t，p，i Refers to the blue color value of pixel i of the target object at time t and position p, AB t，p Refers to the distance weight coefficient at time t and position p, AC t，p Refers to the distance from the target object to the screen at time t and position p;

[0017] Refers to the sum of the color features of the target object;

[0018] The processing process of the feature submodule is as follows: the width value A1 of the target object at time t and position p t，p, the height value A2 of the target object at time t and position p t，p and the distance AC from the target object to the screen at time t and position p t，p Input to the feature submodule and based on the total number of pixels NOC in the target object at time t and position p t，p , output the feature value FPOD of the target object at time t and position p t，p .

[0019] Optionally, the calculation formula of the interaction submodule is as follows:

[0020]

[0021] in:

[0022] INHF t，q Refers to the interaction intention at time t and gesture recognition area q;

[0023] MOC t，q Refers to the total number of gesture features at time t and gesture recognition area q, PPQ t，q，j refers to the weight of gesture feature j at time t and gesture recognition area q, IVB t，q，j refers to the score of gesture feature j at time t and gesture recognition area q, tanh refers to the hyperbolic tangent function, BA t，q Refers to the gesture recognition parameters at time t and gesture recognition area q. 1. BAX t，q Refers to the position value of the gesture at time t and gesture recognition area q, BB t，q refers to gesture recognition parameter 2 at time t and gesture recognition area q;

[0024] The processing process of the interaction submodule is as follows: the feature value FPOD of the target object at time t and position p is t，p Input to the interaction submodule, and based on the total number of gesture features MOC at time t and gesture recognition area q t，q and the score IVB of gesture feature j at time t and gesture recognition area q t，q，j , output the interaction intention INHF at time t and gesture recognition area q t，q .

[0025] Optionally, the calculation formula of the comprehensive submodule is as follows:

[0026]

[0027] in:

[0028] DIRU t refers to the final interaction decision at time t;

[0029] αt Refers to the weight coefficient at time t, β t Refers to the weight coefficient at time t, α t and β t Used to balance the impact of different inputs on decision making, CSD t Refers to the complexity factor of the environment at time t; CSE t refers to the complexity factor 2 of the environment at time t; D1 and D2 both refer to positive numbers to prevent the denominator from being zero;

[0030] ∑ p FPOD t,p refers to the sum of the eigenvalues ​​at time t and all positions p;

[0031] ∑ q INHF t,q Refers to the sum of interaction intentions at time t and all gesture recognition areas q;

[0032] The processing process of the comprehensive submodule is as follows: the feature value FPOD of the target object at time t and position p is t，p and the interaction intention INHF at time t and gesture recognition area q t，q are substituted into the comprehensive submodule, and based on the weight coefficient α at time t t and the weight coefficient β at time t t Output the final interactive decision DIRU at time t t .

[0033] Optionally, the processing process of the iterator module is as follows:

[0034] S1. Set the calculation formula;

[0035]

[0036] S2, set the iteration termination condition;

[0037] Iteration termination condition 1: The number of iterations is 216;

[0038] Iteration termination condition 2: |DIRU t，new -DIRU t，old |<0.0062;

[0039] in:

[0040] PPQ t，q，j，new Refers to the weight of gesture feature j at time t and gesture recognition area q after update;

[0041] PPQ t，q，j，old Refers to the weight of gesture feature j at time t and gesture recognition area q before updating;

[0042] SA refers to the learning rate, which is used to control the step size of weight updates;

[0043] DSD refers to expected interactive decision value;

[0044] DIRU t，new Refers to the final interactive decision at time t after the update, DIRU t，old Refers to the final interaction decision at time t before updating;

[0045] Refers to DIRU t - DSD vs PPQ t，q，j The partial derivative, or gradient, of , indicates that DIRU t At the current weight PPQ t，q，j The downward direction of

[0046] Based on the final interactive decision DIRU at time t t The difference between the expected interactive decision value DSD and the weight PPQ of the gesture feature j at time t and gesture recognition area q before updating is adjusted by the learning rate SA t，q，j，old The degree of change of , output the updated weight PPQ of gesture feature j at time t and gesture recognition area q t，q，j，new , the updated weight PPQ of gesture feature j at time t and gesture recognition area q t，q，j，new The weight PPQ of gesture feature j at time t and gesture recognition area q in the alternative interaction submodule t，q，j , to continuously adjust the interaction intention INHF at time t and gesture recognition area q t，q and the final interactive decision DIRU at time t t , and the loop is terminated by iteration termination condition one and iteration termination condition two.

[0047] Optionally, the extraction module utilizes an image processing algorithm and an infrared sensor to extract a width value of the target object at time t and position p, a height value of the target object at time t and position p, and a distance from the target object to the screen at time t and position p.

[0048] Optionally, the processing module specifically performs preprocessing operations of grayscale conversion, denoising, and contrast enhancement on the collected data.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] 1. The present invention outputs the characteristic value of the target object at time t and position p through the characteristic submodule, thereby realizing a comprehensive analysis of the content displayed on the large screen, providing a solid foundation for subsequent gesture recognition and interaction intention inference. This submodule has significant improvements in accuracy and real-time performance, providing strong support for the intelligent interaction of conference room multimedia terminals. The characteristic value of the target object at time t and position p can accurately capture the characteristics of the target object on the screen, such as position, size, and color. These characteristic values ​​can provide an accurate data basis for subsequent gesture recognition and interaction intention inference, thereby improving the accuracy of the entire system.

[0051] 2. The present invention outputs the interaction intention at time t and gesture recognition area q through the interaction sub-module, and can then accurately infer the user's interaction intention. Through gesture recognition, the user can control the conference room multimedia terminal through simple gesture movements, thereby improving the convenience and efficiency of the interaction. This sub-module can recognize a variety of gesture movements and make corresponding responses based on the user's interaction intention, providing the user with a more natural and intuitive interaction experience. The present invention combines gesture recognition with interaction intention inference to achieve accurate recognition and understanding of the user's gesture movements, providing important technical support for the intelligent interaction of conference room multimedia terminals, and by adjusting the parameters and weights in this sub-module, personalized customization can be achieved. This means that the system can be customized according to the needs and habits of different users, thereby providing an interaction experience that better meets user needs.

[0052] 3. The present invention outputs the final interaction decision at time t through a comprehensive sub-module. This sub-module comprehensively considers multiple input information to make the best interaction decision, which can optimize the entire interaction process and improve the overall performance of the system. Through accurate interaction decision-making, the system can better meet the needs and expectations of users, and improve user satisfaction and loyalty. This sub-module comprehensively considers multiple factors such as target detection and gesture recognition, and realizes accurate grasp and optimization of interaction decisions. This sub-module comprehensively considers the feature values ​​of the target object at time t and position p and the interaction intention at time t and gesture recognition area q to make the final interaction decision, which enables the system to intelligently judge the user's interaction needs and make corresponding responses, thereby improving the level of intelligent interaction.

[0053] 4. The present invention uses an iterator module to t，q，j Iterative update, through the iterative optimization mechanism can be based on DIRU t The value of PPQ is adjusted t，q，j The value of PPQ can be used to optimize the performance of the interaction intention inference model. The weight parameter PPQ can be continuously adjusted through the iterative optimization mechanism. t，q，jThe performance of the interaction intention inference model is continuously improved. Through iteration, the model can be continuously updated and optimized, so that the system can continuously adapt to new touch interaction scenarios and user needs, and improve the flexibility and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of the system method steps based on machine vision large-screen touch interactive conference room multimedia terminal;

[0055] Figure 2 This is a schematic diagram of the overall structure of the multimedia terminal system for large-screen touch interactive conference rooms based on machine vision;

[0056] Figure 3 This is a structural diagram of the recognition and interaction module of the multimedia terminal in the large-screen touch interactive conference room based on machine vision. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The present large-screen touch interactive conference room multimedia terminal based on machine vision is different from the existing multimedia terminal system. The existing multimedia terminal system often finds it difficult to accurately extract the feature values ​​of the target object in the large-screen display content, resulting in limited gesture recognition accuracy. In addition, in the existing interactive recognition, it is often difficult to accurately recognize the user's gesture actions.

[0059] The present large-screen touch interactive conference room multimedia terminal system based on machine vision can accurately calculate the feature value of the target object in the large-screen display content at a specific time or position, and comprehensively considers multiple factors such as the width, height, color value, total number of pixels and distance-related weight coefficient of the target object, thereby realizing accurate extraction of the target object's features. This improvement significantly improves the accuracy of gesture recognition and provides reliable data support for subsequent gesture recognition and interactive decision-making. The present invention uses the extracted feature values ​​and gesture data, combined with a deep learning model for gesture recognition and interactive intention inference, and continuously optimizes model parameters and training data to improve the accuracy of gesture recognition and the intelligence of interactive intention inference. This enables the system to more accurately understand the user's interactive needs and make reasonable decisions.

[0060] For examples, see Figures 1 to 3 This implementation provides a large-screen touch interactive conference room multimedia terminal based on machine vision, including:

[0061] Acquisition module: captures dynamic images in the large-screen touch interaction area and the contact status information between fingers and the screen;

[0062] Among them: The acquisition module uses a camera with high resolution and frame rate to capture dynamic images in the large-screen touch interaction area. The camera should be able to clearly capture the movement trajectory and shape changes of fingers or touch pens on the screen. An infrared sensor array can be installed at the edge of the screen or at a specific location to detect the contact status between the finger or touch pen and the screen. The infrared sensor can provide accurate touch position and contact area information, and is equipped with a high-performance image processor for real-time processing of image data captured by the camera, including image preprocessing, target detection, and feature extraction.

[0063] Extraction module: used to input the dynamic image and the contact state information between the finger and the screen in the large-screen touch interaction area, and output the width value of the target object at time t and position p, the height value of the target object at time t and position p, and the distance from the target object to the screen at time t and position p;

[0064] Processing module: used for inputting the width value of the target object at time t and position p, the height value of the target object at time t and position p, and the distance from the target object to the screen at time t and position p, and the processing module performs data cleaning and sorting on the input parameters;

[0065] Recognition and interaction module: used to input the width value of the target object at time t and position p, the height value of the target object at time t and position p, and the distance from the target object to the screen at time t and position p processed by the processing module, output the feature value of the target object at time t and position p, the interaction intention with the gesture recognition area q at time t, and the final interaction decision at time t, and upload the output result of the interaction module to the management terminal, and the management terminal performs corresponding operations and responses according to the final interaction decision at time t;

[0066] The recognition interaction module includes: feature submodule, interaction submodule, comprehensive submodule and iteration submodule.

[0067] In this embodiment: the media terminal can achieve a more intelligent and personalized interactive experience, and the formulas and methods of the media terminal are universal and flexible, can adapt to different target objects, scenarios and user needs, and support personalized customization and expansion. The media terminal can quickly and accurately respond to user interaction requests, provide a smooth, natural and intuitive interactive experience, thereby improving user satisfaction and loyalty. The multiple sub-modules in the recognition interaction module work together in a system, which can comprehensively analyze the large-screen display content and user gestures to make interactive decisions that best meet user expectations. This highly integrated analysis capability enables the system to provide users with a more intelligent and personalized interactive experience. From target detection to interaction intention inference, and then to the final interaction decision and execution, the entire process is seamlessly connected, reducing the waiting time and operation steps in the traditional interaction method, and improving the interaction efficiency. The present invention can accurately capture the characteristics of the target object on the screen through a precise target detection algorithm, which is used for subsequent gesture recognition and interaction intention inference. The interactive submodule provides a reliable data foundation, and the interactive submodule is interconnected with the feature submodule, which can accurately identify the user's gestures and infer the user's interactive intentions. This highly sensitive gesture recognition capability enables the system to understand the user's interactive needs more accurately. The comprehensive submodule combines the feature values ​​of target detection and the interactive intentions of gesture recognition to make the best interactive decision. This intelligent decision-making system can flexibly adjust the interactive strategy according to different scenarios and user needs, and improve the system's adaptive ability. Multiple submodules are combined together to enable the system to provide users with a smooth, natural and intuitive interactive experience. This experience can reduce the user's cognitive burden and operational difficulty, and improve user satisfaction and comfort. Through intelligent interactive decisions and optimized interactive processes, the system can provide users with efficient and convenient conference operation methods, which helps to reduce waiting time and operational errors in meetings and improve meeting efficiency and quality. Compared with traditional conference room multimedia terminals, the system based on three sets of formulas can achieve a more intelligent and personalized interactive experience. This improvement in the level of intelligence enables the system to better adapt to complex and changing conference environments and user needs. Due to the versatility and flexibility of multiple submodules, the system can adapt to different target objects, scenarios and user needs. At the same time, the system also supports personalized customization and expansion functions, enabling the system to continuously meet users' new needs and new scenarios.

[0068] See also Figures 1 to 3 , the feature submodule processing process is as follows:

[0069]

[0070]

[0071] in:

[0072] FPOD t，p Refers to the feature value of the target object at time t and position p;

[0073] A1 t，p Refers to the width of the target object at time t and position p, A2 t，p Refers to the height value of the target object at time t and position p, A1 t，p and A2 t，p Refers to the size used to describe the target object;

[0074] NOC t，p Refers to the total number of pixels in the target object at time t and position p, NOC t，p Used to indicate the size or complexity of the target object;

[0075] YAS t，p，i Refers to the red color value of pixel i of the target object at time t and position p, YBS t，p，i Refers to the green color value of pixel i of the target object at time t and position p, YCS t，p，i Refers to the blue color value of pixel i of the target object at time t and position p, YAS t，p，i , YBS t，p，i 、YCS t，p，i These three parameters are used to describe the color characteristics of the target object;

[0076] AB t，p Refers to the distance weight coefficient at time t and position p, AB t，p Used to adjust the effect of the distance from the target object to the center of the screen on the feature value;

[0077] AC t，p Refers to the distance from the target object to the screen at time t and position p, AC t，p Used to describe the location of the target object on the screen;

[0078] Refers to the sum of the color features of the target object;

[0079] The processing of the feature submodule is as follows: the width value A1 of the target object at time t and position p t，p , the height value A2 of the target object at time t and position p t，p and the distance AC from the target object to the screen at time t and position p t，p Input to the feature submodule and based on the total number of pixels NOC in the target object at time t and position p t，p , output the feature value FPOD of the target object at time t and position p t，p .

[0080] In this embodiment: the present invention can quickly and accurately locate the target object in the content displayed on the large screen, and extract its feature values ​​at a specific time or position, which include the size of the target object, i.e., width, height, color information red, green, blue color values, and distance to the center of the screen, etc., providing rich data support for subsequent gesture recognition and interaction intention inference, and through precise target detection, the accuracy of subsequent gesture recognition and interaction intention inference can be ensured, thereby improving the stability and reliability of the entire system, and fast and accurate target detection can reduce the response time of the system, so that users can experience a more smooth and natural interaction experience, and the feature extraction of this submodule is universal, can be applied to different target objects and scenarios, and enhances the flexibility and scalability of the system;

[0081] This submodule realizes a comprehensive analysis of the large-screen display content, providing a solid foundation for subsequent gesture recognition and interaction intention inference. Compared with traditional target detection methods, this submodule has significant improvements in accuracy and real-time performance, providing strong support for the intelligent interaction of conference room multimedia terminals, and FPOD t，p It can accurately capture the features of the target objects on the screen, such as position, size, color, etc. These feature values ​​provide an accurate data basis for subsequent gesture recognition and interaction intention inference, thereby improving the accuracy of the entire system.

[0082] See also Figures 1 to 3 , the interaction submodule processing process is as follows:

[0083]

[0084] in:

[0085] INHF t，q Refers to the interaction intention at time t and gesture recognition area q, INHF t，q It represents the user interaction intention inferred by the system based on the gesture features and the feature values ​​of the target object;

[0086] MOC t，q Refers to the total number of gesture features at time t and gesture recognition area q, MOC t，q represents the number of gesture features recognized in the gesture recognition area q;

[0087] PPQ t，q，j Refers to the weight of gesture feature j at time t and gesture recognition area q, PPQ t，q，j Used to adjust the impact of gesture feature j on interaction intention inference;

[0088] IVB t，q，j Refers to the score of gesture feature j at time t and gesture recognition area q, IVBt，q，j represents the recognition score or importance of gesture feature j;

[0089] tanh refers to the hyperbolic tangent function;

[0090] BA t，q refers to the gesture recognition parameter 1 at time t and gesture recognition area q, BA t，q Used to describe the characteristics of gesture recognition models;

[0091] BAX t，q Refers to the position value of the gesture at time t and gesture recognition area q;

[0092] BB t，q Refers to the gesture recognition parameter 2 at time t and gesture recognition area q, BB t，q Also used to describe the characteristics of gesture recognition models;

[0093] The processing of the interaction submodule is as follows: the feature value FPOD of the target object at time t and position p is t，p Input to the interaction submodule, and based on the total number of gesture features MOC at time t and gesture recognition area q t，q and the score IVB of gesture feature j at time t and gesture recognition area q t，q，j , output the interaction intention INHF at time t and gesture recognition area q t，q .

[0094] In this embodiment: This submodule uses the feature value FPOD of the target object at time t and position p in the feature submodule t，p And gesture data to identify the user's gestures, and infer the interaction intention at a specific time or location. Through the weighted summation and normalization of gesture features, as well as consideration of key angles or position parameters, this submodule can accurately infer the user's interaction intention. Through gesture recognition, users can control the conference room multimedia terminal with simple gestures, which improves the convenience and efficiency of interaction. This submodule can recognize a variety of gestures and respond accordingly according to the user's interaction intention, providing users with a more natural and intuitive interaction experience. This submodule combines gesture recognition with interaction intention inference to achieve accurate recognition and understanding of user gestures, providing important technical support for the intelligent interaction of conference room multimedia terminals. Compared with traditional gesture recognition methods, this submodule has significantly improved recognition accuracy and robustness, and can better adapt to complex and changing interaction scenarios. In this submodule, based on FPOD t，p The accurate extraction of the user's gestures can accurately identify the user's gestures, which enables the system to accurately understand the user's interaction intentions and make corresponding responses, improving the accuracy and convenience of the interaction.

[0095] For INHF t，q It can recognize a variety of gesture actions, such as clicking, sliding, zooming, etc., which provides users with a variety of interaction methods, allowing users to choose the most suitable interaction method according to their habits and needs, and by adjusting the parameters and weights in this submodule, such as the weights of gesture features, personalized customization can be achieved. This means that the system can be customized according to the needs and habits of different users, thereby providing an interactive experience that is more in line with user needs.

[0096] See also Figures 1 to 3 , the processing of the comprehensive submodule is as follows:

[0097]

[0098] in:

[0099] DIRU t refers to the final interaction decision at time t;

[0100] α t Refers to the weight coefficient at time t, β t Refers to the weight coefficient at time t, α t and β t Used to balance the impact of different inputs on decision making, CSD t Refers to the complexity factor of the environment at time t; CSE t refers to the complexity factor 2 of the environment at time t; D1 and D2 both refer to positive numbers to prevent the denominator from being zero;

[0101] ∑ p FPOD t,p refers to the sum of the eigenvalues ​​at time t and all positions p;

[0102] ∑ q INHF t,q Refers to the sum of interaction intentions at time t and all gesture recognition areas q;

[0103] The processing of the comprehensive submodule is as follows: the feature value FPOD of the target object at time t and position p is t，p and the interaction intention INHF at time t and gesture recognition area q t，q are substituted into the comprehensive submodule, and based on the weight coefficient α at time t t and the weight coefficient β at time t t Output the final interactive decision DIRU at time t t .

[0104] In this embodiment: this submodule combines the eigenvalues ​​of target detection and the interaction intention of gesture recognition to make the final interaction decision. By weighted summation and normalization of different inputs, as well as consideration of the differences between eigenvalues ​​and interaction intentions, this submodule can derive the interaction decision that best meets the user's expectations. This submodule can comprehensively consider a variety of input information and make the best interaction decision, thereby optimizing the entire interaction process and improving the overall performance of the system. Through accurate interaction decisions, the system can better meet the needs and expectations of users and improve user satisfaction and loyalty.

[0105] This submodule takes into account multiple factors such as target detection and gesture recognition, and realizes the accurate grasp and optimization of interactive decision-making. Compared with the traditional interactive decision-making method, this submodule has significantly improved the accuracy and flexibility of decision-making, and can better adapt to complex and changing interactive scenarios and user needs. This submodule integrates FPOD t，p and INHF t，q To make the final interaction decision, the system can intelligently judge the user's interaction needs and respond accordingly, improving the level of intelligent interaction. t The generation of DIRU takes into account a variety of factors, including the characteristics of the target object, the user's gestures, etc., which enables the system to optimize the interaction process, reduce unnecessary operation steps and waiting time, and improve interaction efficiency. Through intelligent interaction decisions and optimized interaction processes, DIRU t It can provide users with a smoother, more natural and intuitive interactive experience, help improve user satisfaction and loyalty, and lay a good foundation for the promotion and application of conference room multimedia terminals.

[0106] See also Figures 1 to 3 , the iterator module processing process is as follows:

[0107] S1. Set the calculation formula;

[0108]

[0109] S2, set the iteration termination condition;

[0110] Iteration termination condition 1: The number of iterations is 216;

[0111] Iteration termination condition 2: |DIRU t，new -DIRU t，old |<0.0062;

[0112] in:

[0113] PPQ t，q，j，new Refers to the weight of gesture feature j at time t and gesture recognition area q after update;

[0114] PPQ t，q，j，old Refers to the weight of gesture feature j at time t and gesture recognition area q before updating;

[0115] SA refers to the learning rate, which is used to control the step size of weight updates;

[0116] DSD refers to expected interactive decision value;

[0117] DIRU t，new Refers to the final interactive decision at time t after the update, DIRU t，old Refers to the final interaction decision at time t before updating;

[0118] Refers to DIRU t - DSD vs PPQ t，q，j The partial derivative, or gradient, of , indicates that DIRU t At the current weight PPQ t，q，j The downward direction of

[0119] Based on the final interactive decision DIRU at time t t The difference between the expected interactive decision value DSD and the weight PPQ of the gesture feature j at time t and gesture recognition area q before updating is adjusted by the learning rate SA t，q，j，old The degree of change of , output the updated weight PPQ of gesture feature j at time t and gesture recognition area q t，q，j，new , the updated weight PPQ of gesture feature j at time t and gesture recognition area q t，q，j，new The weight PPQ of gesture feature j at time t and gesture recognition area q in the alternative interaction submodule t，q，j , to continuously adjust the interaction intention INHF at time t and gesture recognition area q t，q and the final interactive decision DIRU at time t t , and the loop is terminated by iteration termination condition one and iteration termination condition two.

[0120] In this embodiment: the multimedia terminal uses a high-definition camera to capture the image of the finger touching the screen and accurately locates the finger. This machine vision-based technology makes the touch interaction more accurate and sensitive, and the final interaction decision DIRU at time t is made by integrating the calculation results of the submodules. t To iterate the weight PPQ of gesture feature j at time t and gesture recognition area q in the interaction submodule t，q，j, to achieve continuous optimization of the interaction intention inference model. This iterative mechanism enables the model to continuously adapt to new touch interaction scenarios and improve the accuracy and stability of interaction. This multimedia terminal integrates conference audio and video terminal equipment, high-definition cameras, array microphones, hidden speakers and other modules, and realizes the integration of multiple functions such as audio and video communication, touch interaction, and multimedia display. Through precise touch positioning and optimized interaction intention inference models, users can get a smoother and more natural interaction experience. The iterative optimization mechanism enables the system to continuously adapt to new touch interaction scenarios and user needs, improving the flexibility and scalability of the system. The highly integrated multimedia terminal reduces the number and complexity of equipment and reduces the maintenance cost of the system. Through iterative PPQ t，q，j This parameter determines the importance of gesture features in the interaction intention inference process;

[0121] Through the iterative optimization mechanism, we can t The value of PPQ is adjusted t，q，j The value of , thereby optimizing the performance of the interactive intent inference model. Specifically, when DIRU t If it is large, it means that the performance of the current model may be poor, and we need to adjust PPQ t，q，j To reduce DIRU t value; when DIRU t When it is smaller, it means that the performance of the current model is good, and we can continue to fine-tune PPQ t，q，j To maintain the stability of the model, we can continuously adjust the weight parameter PPQ through the iterative optimization mechanism. t，q，j , which continuously improves the performance of the interaction intent inference model. Through the iterative optimization mechanism, we can achieve continuous updating and optimization of the model.

[0122] In the specific implementation process, the various sub-modules in this method are used to form a multimedia terminal system based on machine vision large-screen touch interactive conference room, and the width value A1 of the target object at time t and position p is t，p , the height value A2 of the target object at time t and position p t，p and the distance AC from the target object to the screen at time t and position p t，p Input to the feature submodule, output the feature value FPOD of the target object at time t and position p t，p This submodule realizes the comprehensive analysis of the large-screen display content, providing a solid foundation for subsequent gesture recognition and interaction intention inference. Compared with the traditional target detection method, this submodule has significant improvements in accuracy and real-time performance, providing strong support for the intelligent interaction of conference room multimedia terminals, and FPOD t，pIt can accurately capture the characteristics of the target object on the screen, such as position, size and color. These characteristic values ​​can provide an accurate data basis for subsequent gesture recognition and interaction intention inference, thereby improving the accuracy of the entire system;

[0123] The feature value FPOD of the target object at time t and position p t，p Input to the interaction submodule, output the interaction intention INHF at time t and gesture recognition area q t，q ,This submodule can accurately infer the user's interaction intention. Through gesture recognition, the user can control the conference room multimedia terminal through simple gestures, thereby improving the convenience and efficiency of interaction. This submodule can recognize a variety of gestures and respond accordingly according to the user's interaction intention, providing users with a more natural and intuitive interaction experience. This submodule combines gesture recognition with interaction intention inference to achieve accurate recognition and understanding of user gestures, which can provide important technical support for the intelligent interaction of conference room multimedia terminals. And by adjusting the parameters and weights in this submodule, such as the weight of gesture features, personalized customization can be achieved. This means that the system can be customized according to the needs and habits of different users, thereby providing an interaction experience that better meets user needs;

[0124] The feature value FPOD of the target object at time t and position p t，p and the interaction intention INHF at time t and gesture recognition area q t，q are all substituted into the comprehensive submodule to output the final interactive decision DIRU at time t t This submodule can comprehensively consider multiple input information to make the best interactive decision, thereby optimizing the entire interactive process and improving the overall performance of the system. Through accurate interactive decision-making, the system can better meet the needs and expectations of users to improve user satisfaction and loyalty. This submodule comprehensively considers multiple factors such as target detection and gesture recognition to achieve accurate grasp and optimization of interactive decisions. This submodule integrates FPOD t，p and INHF t，q To make the final interaction decision, the system can intelligently judge the user's interaction needs and respond accordingly, improving the level of intelligent interaction;

[0125] DIRU based on iterator module t PPQ t，q，j Iterative update, through the iterative optimization mechanism, can be based on DIRU t The value of PPQ is adjusted t，q，j The value of PPQ can be used to optimize the performance of the interaction intention inference model. The weight parameter PPQ can be continuously adjusted through the iterative optimization mechanism. t，q，j, so that the performance of the interaction intention inference model is continuously improved. The model can be continuously updated and optimized through iteration. The iterative optimization mechanism enables the system to continuously adapt to new touch interaction scenarios and user needs, improving the flexibility and scalability of the system;

[0126] This allows the various sub-modules to cooperate with each other in calculations, and to perform overall cycles and iterations, so that the overall system has the effect of automatic optimization and updating, and thus better adaptability.

[0127] See also Figure 1 , Figure 2 and Figure 3 , the extraction module uses an image processing algorithm and an infrared sensor to extract a width value of a target object at time t and position p, a height value of a target object at time t and position p, and a distance from the target object to the screen at time t and position p;

[0128] The processing module specifically performs preprocessing operations such as grayscale conversion, denoising, and contrast enhancement on the collected data to improve image quality.

[0129] In this embodiment: the extraction module uses an image processing algorithm, which can be edge detection, contour extraction, etc., to identify the shape of the finger or touch pen in the captured image and calculate its width and height, and the extraction module uses an image processing algorithm to calculate the Euclidean distance of the target object, that is, the finger or touch pen, to the center of the screen. The data is extracted based on the extraction module and then processed by the processing module, and can be directly input into the recognition interaction module for subsequent processing.

[0130] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Based on machine vision, the large-screen touch interactive conference room multimedia terminal is characterized by: include: Acquisition module: captures dynamic images in the large-screen touch interaction area and the contact status information between fingers and the screen; Extraction module: used to input the dynamic image and the contact state information between the finger and the screen in the large-screen touch interaction area, and output the width value of the target object at time t and position p, the height value of the target object at time t and position p, and the distance from the target object to the screen at time t and position p; Processing module: used for inputting the width value of the target object at time t and position p, the height value of the target object at time t and position p, and the distance from the target object to the screen at time t and position p, and the processing module performs data cleaning and sorting on the input parameters; Identification and interaction module: used to input the width value of the target object at time t and position p, the height value of the target object at time t and position p, and the distance from the target object to the screen at time t and position p after being processed by the processing module, output the feature value of the target object at time t and position p, the interaction intention with the gesture recognition area q at time t, and the final interaction decision at time t, and upload the output results of the interaction module to the management terminal, which performs corresponding operations and responses according to the final interaction decision at time t.

2. The large-screen touch interactive conference room multimedia terminal based on machine vision according to claim 1 is characterized by: The identification interaction module includes: a feature submodule, an interaction submodule, a comprehensive submodule and an iteration submodule.

3. The large-screen touch interactive conference room multimedia terminal based on machine vision according to claim 2 is characterized by: The calculation formula of the feature submodule is as follows: in: FPOD t，p Refers to the feature value of the target object at time t and position p; A1 t，p Refers to the width of the target object at time t and position p, A2 t，p Refers to the height value of the target object at time t and position p, NOC t，p Refers to the total number of pixels in the target object at time t and position p, YAS t，p，i Refers to the red color value of pixel i of the target object at time t and position p, YBS t，p，i Refers to the green color value of pixel i of the target object at time t and position p, YCS t，p，i Refers to the blue color value of pixel i of the target object at time t and position p, AB t，p Refers to the distance weight coefficient at time t and position p, AC t，p Refers to the distance from the target object to the screen at time t and position p; Refers to the sum of the color features of the target object; The processing process of the feature submodule is as follows: the width value A1 of the target object at time t and position p t，p , the height value A2 of the target object at time t and position p t，p and the distance AC from the target object to the screen at time t and position p t，p Input to the feature submodule and based on the total number of pixels NOC in the target object at time t and position p t，p , output the feature value FPOD of the target object at time t and position p t，p .

4. The large-screen touch interactive conference room multimedia terminal based on machine vision according to claim 3 is characterized by: The calculation formula of the interaction submodule is as follows: in: INHF t，q Refers to the interaction intention at time t and gesture recognition area q; MOC t，q Refers to the total number of gesture features at time t and gesture recognition area q, PPQ t，q，j refers to the weight of gesture feature j at time t and gesture recognition area q, IVB t，q，j refers to the score of gesture feature j at time t and gesture recognition area q, tanh refers to the hyperbolic tangent function, BA t，q Refers to the gesture recognition parameters at time t and gesture recognition area q.

1. BAX t，q Refers to the position value of the gesture at time t and gesture recognition area q, BB t，q refers to gesture recognition parameter 2 at time t and gesture recognition area q; The processing process of the interaction submodule is as follows: the feature value FPOD of the target object at time t and position p is t，p Input to the interaction submodule, and based on the total number of gesture features MOC at time t and gesture recognition area q t，q and the score IVB of gesture feature j at time t and gesture recognition area q t，q，j , output the interaction intention INHF at time t and gesture recognition area q t，q .

5. The large-screen touch interactive conference room multimedia terminal based on machine vision according to claim 4 is characterized by: The calculation formula of the comprehensive submodule is as follows: in: DIRU t refers to the final interaction decision at time t; α t Refers to the weight coefficient at time t, β t Refers to the weight coefficient at time t, α t and β t Used to balance the impact of different inputs on decision making, CSD t Refers to the complexity factor of the environment at time t; CSE t refers to the complexity factor 2 of the environment at time t; D1 and D2 both refer to positive numbers to prevent the denominator from being zero; ∑ p FPOD t,p refers to the sum of the eigenvalues ​​at time t and all positions p; ∑ q INHF t,q Refers to the sum of interaction intentions at time t and all gesture recognition areas q; The processing process of the comprehensive submodule is as follows: the feature value FPOD of the target object at time t and position p is t，p and the interaction intention INHF at time t and gesture recognition area q t，q are substituted into the comprehensive submodule, and based on the weight coefficient α at time t t and the weight coefficient β at time t t Output the final interactive decision DIRU at time t t .

6. The large-screen touch interactive conference room multimedia terminal based on machine vision according to claim 5 is characterized by: The processing process of the iterator module is as follows: S1. Set the calculation formula; S2, set the iteration termination condition; Iteration termination condition 1: The number of iterations is 216; Iteration termination condition 2: |DIRU t，new -DIRU t，old |<0.0062; in: PPQ t，q，j，new Refers to the weight of gesture feature j at time t and gesture recognition area q after update; PPQ t，q，j，old Refers to the weight of gesture feature j at time t and gesture recognition area q before updating; SA refers to the learning rate, which is used to control the step size of weight updates; DSD refers to expected interactive decision value; DIRU t，new Refers to the final interactive decision at time t after the update, DIRU t，old refers to the final interaction decision at time t before updating; Refers to DIRU t - DSD vs PPQ t，q，j The partial derivative, or gradient, of , indicates that DIRU t At the current weight PPQ t，q，j The downward direction of Based on the final interactive decision DIRU at time t t The difference between the expected interactive decision value DSD and the weight PPQ of the gesture feature j at time t and gesture recognition area q before updating is adjusted by the learning rate SA t，q，j，old The degree of change of , output the updated weight PPQ of gesture feature j at time t and gesture recognition area q t，q，j，new , the updated weight PPQ of gesture feature j at time t and gesture recognition area q t，q，j，new The weight PPQ of gesture feature j at time t and gesture recognition area q in the alternative interaction submodule t，q，j , to continuously adjust the interaction intention INHF at time t and gesture recognition area q t，q and the final interactive decision DIRU at time t t , and the loop is terminated by iteration termination condition one and iteration termination condition two.

7. The large-screen touch interactive conference room multimedia terminal based on machine vision according to claim 1 is characterized by: The extraction module utilizes image processing algorithms and infrared sensors to extract the width value of the target object at time t and position p, the height value of the target object at time t and position p, and the distance from the target object to the screen at time t and position p.

8. The large-screen touch interactive conference room multimedia terminal based on machine vision according to claim 1 is characterized by: The processing module specifically performs preprocessing operations of grayscale conversion, noise removal, and contrast enhancement on the collected data.

Images