Self-adaptive feedback regulation and control system and method for teleprompter

Abstract

The invention discloses a self-adaptive feedback regulation and control system and method for a teleprompter, and relates to the technical field of big data analysis, and aims to ensure that the importance of related contents is evaluated only when a user concentrates attention by accurately identifying the effective gaze of the user on the teleprompter and combining data such as a sight line deviation angle and gaze time, so that the user experience is improved. The distribution of information display resources is improved, excessive display of skilled information is avoided, and the interactivity and the intelligent level of the teleprompter are improved. Meanwhile, the importance degree of the display content is calculated, the content concerned by the user is dynamically screened and preferentially highlighted in combination with the content similarity, it is ensured that key information is displayed in real time, and the efficiency and practicability of the teleprompter are improved. The display progress is dynamically monitored in combination with the voice recognition technology, the display rhythm is accurately adjusted, it is ensured that the most concerned content of the user is displayed at the appropriate moment, the display process is optimized, and the application value and user experience of the teleprompter are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and in particular to an adaptive feedback control system and method for a teleprompter. Background Art

[0002] Teleprompters are of great significance in various performances and speeches, and can effectively improve the expressiveness and confidence of performers. By displaying text in real time, it helps speakers, singers or hosts avoid forgetting words or missing key information, thereby ensuring the fluency and accuracy of the speech or performance content. The teleprompter can also dynamically adjust the displayed text according to the rhythm and content requirements of the performance, improving the efficiency and effectiveness of information transmission. Especially in occasions that require quick response and efficient performance, the use of teleprompters can greatly reduce the burden on performers, improve the overall performance quality, and ensure the viewing experience of the audience on site.

[0003] Existing teleprompters only adjust the scrolling speed and dwell time of displayed content according to the user's performance progress, and fail to effectively analyze user needs and prompt content that may be forgotten in advance, resulting in breakpoints or unsmooth performance. Existing technologies lack the function of displaying key content that users are prone to forget in advance, and also do not make dynamic adjustments through intelligent analysis of user performance, thus failing to achieve efficient display resource allocation, resulting in users not getting enough preparation time when they need prompts. This lack of intelligent adaptation limits the auxiliary effect of the teleprompter in performance, affecting the coherence and naturalness of the user's performance. Summary of the invention

[0004] The object of the present invention is to provide an adaptive feedback control system and method for a teleprompter to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: an adaptive feedback control method for a teleprompter, the adaptive feedback control method comprising the following steps:

[0006] Step S1, collecting the user's facial feature data, and analyzing the user's facial orientation based on the facial feature data, and marking the moment when the user meets the facial orientation condition as the key marking moment;

[0007] The user's facial feature data is collected through an image sensor, and the facial feature data includes the user's eye position, nose position and mouth position. The facial feature data is input into a facial recognition algorithm, and the user's facial orientation is analyzed. Based on the user's facial orientation analysis result, it is determined whether the user is facing the teleprompter display screen. When the user's face is facing the teleprompter display screen, the facial orientation condition is met, and the moment when the facial orientation condition is met is marked as the key marked moment.

[0008] The technology for identifying facial orientation is mainly achieved through facial key point detection and head posture estimation. By detecting the position of facial key points and combining algorithms such as EPnP or PnP, the pitch, yaw and roll angles of the head can be inferred. Deep learning models and 3D sensors such as depth cameras can also directly analyze facial orientation.

[0009] The user's facial feature data is collected through image sensors, and the user's facial orientation is analyzed to ensure that the teleprompter content is accurately displayed during the user's attention period.

[0010] Step S2, collecting the user's eye data at the key marking moment, wherein the user's eye data includes the user's pupil position data and corneal reflection position data, and calculating the user's line of sight deviation angle in combination with the user's eye data;

[0011] Step S2-1, with the ground as a reference, construct a three-dimensional space coordinate system with the geometric center of the teleprompter display screen as the origin, and define the directions as follows: the X-axis is parallel to the ground and extends along the horizontal side of the display screen, with the positive direction being from the center to the right and the negative direction being to the left; the Y-axis is perpendicular to the ground and extends along the vertical side of the display screen, with the positive direction being upward and the negative direction being downward; the Z-axis is perpendicular to the plane of the display screen, with the positive direction being from the center to the direction where the user is located and the negative direction being toward the back of the display screen, and using an infrared sensor to collect the user's eye data at the key marking moment, the user's eye data includes the user's pupil position data and the corneal reflection position data; when the user is at the key marking moment, the user's face faces the teleprompter display screen, and the infrared sensor can completely obtain the user's eye data; at any key marking moment, the origin position of the space coordinate system is P0(x0, y0, z0), the position of the pupil position in the space coordinate system is Ppupil(xk, yk, zk), and the position of the corneal reflection position in the space coordinate system is Pcornea(xc, yc, zc);

[0012] Step S2-2, calculate the user's line of sight deviation angle according to the user's eye data at the key marking moment, the pupil position includes the left eye pupil position and the right eye pupil position, the position of the left eye pupil position in the spatial coordinate system is Ppupil, left (xk, l, yk, l, zk, l), the position of the right eye pupil position in the spatial coordinate system is Ppupil, right (xk, r, yk, r, zk, r); the corneal reflection position includes the left eye corneal reflection position and the right eye corneal reflection position, the left eye corneal reflection position The position in the spatial coordinate system is Pcornea,left(xc,l,yc,l,zc,l), the position of the right eye corneal reflection position in the spatial coordinate system is Pcornea,right(xc,r,yc,r,zc,r), the pupil position and corneal reflection position of the left eye are substituted into the line of sight deviation angle calculation formula to obtain the line of sight deviation angle of the left eye, the pupil position and corneal reflection position of the right eye are substituted into the line of sight deviation angle calculation formula to obtain the line of sight deviation angle of the right eye, and the line of sight deviation angle calculation formula is as follows:

[0013]

[0014] Where Wt represents the user's line of sight deviation angle at the time t of the key mark.

[0015] The corneal reflection position refers to the position where the light source is reflected from the surface of the eye, and is usually determined by the position of the light spot reflected from the corneal surface of the eye.

[0016] The line of sight deviation angle is expressed as the angle between the user's line of sight and the center line of the eyeball (or the vertical eye line).

[0017] Step S3, calculating the user's sight focus position at the time of key marking through the user's eye data, judging whether the user's gaze on the teleprompter display screen at the time of key marking is effective according to the sight focus position, storing the user's sight deviation angle and gaze time during effective gaze and constructing a user attention data set;

[0018] Step S3-1, calculating the user's sight focus position at the time of key marking according to the user's eye data, the sight focus position calculation process is as follows:

[0019] Step S3-1-1, the pupil position and corneal reflection position of the left eye are brought into the sight direction vector calculation formula to obtain the sight direction vector of the left eye, and the pupil position and corneal reflection position of the right eye are brought into the sight direction vector calculation formula to obtain the sight direction vector of the right eye, the sight direction vector calculation formula is as follows:

[0020] V=(xc-xk, yc-yk, zc-zk);

[0021] Where V is the user's sight direction vector, specifically, the direction from the pupil position of the eye to the object through the corneal reflection point position;

[0022] Step S3-2-2, calculate the sight focus position according to the user's eye data and the sight direction vector V, the sight focus position is represented as the intersection of the left eye sight direction vector and the right eye sight direction vector, the calculation process is as follows:

[0023] In the line of sight of the left eye, any point Rl(tl) is expressed as:

[0024] Rl(tl)=Ppupil,left+tl*Vl;

[0025] Wherein, Rl(tl) represents a point in the sight direction of the left eye; tl represents a scalar parameter, specifically the distance between point Rl and the left eye pupil position; Vl represents the sight direction vector of the left eye;

[0026] In the line of sight of the right eye, any point Rr(tr) is expressed as:

[0027] Rr(tr)=Ppupil,right+tr*Vr;

[0028] Where Rr(tr) is a point in the direction of the left eye's sight; tr is a scalar parameter, specifically the distance between point Rr and the pupil position of the right eye; Vr is the sight direction vector of the right eye;

[0029] The focus position of the sight is calculated according to the formula of equal intersection coordinates. The calculation formula is as follows:

[0030] Rl(tl)=Rr(tr);

[0031] According to the condition of equal intersection position, three equations are obtained by respectively substituting the x-axis, y-axis and z-axis. The equations are solved to obtain the scalar parameters tl and tr. The values ​​of tl and tr are substituted to obtain the coordinates of the intersection, which is the focus position of the user's sight at the time of key marking. The coordinates are expressed as (xt, yt, zt);

[0032] Step S3-3: judging whether the user's gaze on the teleprompter display screen at the key marking moment is valid based on the calculated user's sight focus position, and the judging process is as follows:

[0033] Obtain the edge coordinate data of the effective display area of the teleprompter display screen on the x-axis and y-axis. The plane effective display area is defined as the area on the plane formed by the x-axis and y-axis where the user can effectively gaze at and focus on the content displayed on the teleprompter display screen. The edge coordinate data includes the maximum value xmax and minimum value xmin of the display screen on the x-axis, and the maximum value ymax and minimum value ymin on the y-axis. Determine whether the line-of-sight focus position is within the plane effective display area based on the edge coordinate data. The determination process is as follows:

[0034] When xmax ≥ xt ≥ xmin and ymax ≥ yt ≥ ymin, it is determined that the user's line-of-sight focus is located within the plane effective display area;

[0035] When the x-axis coordinate of the focus does not satisfy xmax ≥ xt ≥ xmin, or the y-axis coordinate of the focus does not satisfy ymax ≥ yt ≥ ymin, it is determined that the user's line-of-sight focus is not within the plane effective display area;

[0036] Set the focus error threshold Zmax, and judge the vertical distance from the line-of-sight focus position to the teleprompter display screen based on the z-axis coordinate data of the line-of-sight focus position. The judgment process is as follows:

[0037] When |zt - z0| ≤ Zmax, it is determined that the line-of-sight focus position satisfies effective gazing in terms of vertical distance;

[0038] When |zt - z0| > Zmax, it is determined that the line-of-sight focus position does not satisfy effective gazing in terms of vertical distance;

[0039] When the line-of-sight focus position simultaneously satisfies the conditions of being within the plane effective display area and satisfying effective gazing in terms of vertical distance, it is determined that the user's gaze at the teleprompter display screen is an effective gaze; when the line-of-sight focus position does not satisfy being within the plane effective display area, or does not satisfy the condition of effective gazing in terms of vertical distance, it is determined that the user's gaze at the teleprompter display screen is an ineffective gaze;

[0040] Step S3-4: Obtain the gaze time T from the moment when the user's line-of-sight focus first enters the effective display area until the moment when the user's line-of-sight focus leaves this area. Store the user's line-of-sight deviation angle and gaze time to construct a user attention data set.

[0041] By calculating the line-of-sight focus position and judging effective gazing, it is possible to accurately identify whether the user is focusing on the teleprompter display screen. At the same time, by combining the line-of-sight deviation angle and gaze time, monitor the user's attention level and optimize the usage experience.

[0042] Step S4, setting a content similarity threshold, calculating the importance of the content currently displayed on the teleprompter display screen according to the user attention data set, analyzing and screening the undisplayed content in combination with the content similarity threshold, and highlighting the analyzed and screened content;

[0043] Step S4-1, set the content importance threshold Imin and the content similarity threshold Umin, and calculate the importance of the content currently displayed on the teleprompter display screen through the user attention data set. The importance calculation formula is as follows:

[0044]

[0045] Where, Icontent represents the importance of the content of effective gaze; ET represents the weighting coefficient of gaze time; T represents the gaze time; EW represents the weighting coefficient of the sight deviation angle; Wmax represents the maximum value of the sight deviation angle, specifically the maximum deviation angle received by the teleprompter display screen;

[0046] Step S4-2, filter out the content with Icontent≥Imin, and use the filtered content as reference content, input the undisplayed content and the reference content into the neural network to calculate the content similarity, and highlight the undisplayed content with content similarity greater than Umin according to the calculation result.

[0047] By calculating the importance and similarity of the content, the system can prioritize the content that users are most concerned about. Combined with highlighting the related content that is not displayed, the interaction effect and efficiency between the user and the teleprompter can be improved.

[0048] Step S5, using voice recognition technology to monitor the content display progress of the teleprompter display screen in real time, and adjusting the highlighted content to be displayed with priority according to the content display progress. The priority display adjustment means that during the content display process of the teleprompter, the highlighted content is adjusted to be displayed with priority.

[0049] Set the content progress threshold Lmax, monitor the content display progress Lcurrent of the teleprompter display screen in real time through voice recognition technology, and identify the content that the teleprompter needs to display next, and judge whether it is the highlighted content. When the content that needs to be displayed next is the highlighted content, judge the progress of the currently displayed content. The judgment process is as follows:

[0050] When Lcurrent ≥ Lmax, adjust the content of the teleprompter display screen to the content highlighted in the next step;

[0051] When Lcurrent<Lmax, the contents of the teleprompter display screen continue to be displayed.

[0052] By monitoring the content display progress in real time, the system can prioritize the highlighted content based on the voice recognition results, ensuring that the content that the user is interested in is presented in a timely manner, thereby optimizing the display rhythm of the teleprompter.

[0053] Further, an adaptive feedback control system for a teleprompter, the adaptive feedback control system comprising a facial analysis module, a sight line calculation module, a focus determination module, a content analysis module and a display control module;

[0054] The facial analysis module is used to analyze whether the user is facing the teleprompter through an image sensor and mark key moments; the sightline calculation module is used to calculate the sightline deviation angle based on the user's eye data to determine whether the user's sight is deviating from the teleprompter display screen; the focus judgment module is used to determine whether the user's sight is concentrated in the effective display area and whether the gaze is effective; the content analysis module is used to calculate the importance of the content that the user is paying attention to and highlight it according to the similarity; the display control module is used to adjust the display content according to the progress of voice recognition to ensure that the highlighted content is displayed in advance;

[0055] The output end of the facial analysis module is electrically connected to the input end of the sight line calculation module; the output end of the sight line calculation module is electrically connected to the input end of the focus judgment module; the output end of the focus judgment module is electrically connected to the input end of the content analysis module, and the output end of the content analysis module is electrically connected to the input end of the display control module;

[0056] The facial analysis module includes an image acquisition unit and a direction analysis unit; the image acquisition unit is used to acquire user facial image data; the direction analysis unit is used to analyze whether the user is facing the teleprompter;

[0057] The sight line calculation module includes an infrared acquisition unit and a deviation calculation unit; the infrared acquisition unit is used to collect user eye data; the deviation calculation unit is used to calculate the deviation angle of the sight line;

[0058] The focus judgment module includes a focus calculation unit and a gaze judgment unit; the focus calculation unit is used to calculate the user's line of sight focus position; the gaze judgment unit is used to judge whether the user's gaze is valid;

[0059] The content analysis module includes an importance calculation unit and a similarity analysis unit; the importance calculation unit is used to calculate the importance of the content; the similarity analysis unit is used to analyze the similarity between the undisplayed content and the reference content;

[0060] The display control module includes a progress monitoring unit and a priority display unit; the progress monitoring unit is used to monitor the content display progress of the teleprompter in real time; the priority display unit is used to adjust the highlighted content according to the progress.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. The present invention starts by accurately identifying the user's effective gaze on the teleprompter, and through data such as eye position, sight deviation angle and gaze time, ensures that the importance of relevant content is evaluated only when the user is paying attention, thereby improving the allocation of information display resources. This method effectively avoids the display time occupation of invalid information and improves the interactivity and intelligence level of the teleprompter.

[0063] 2. The present invention calculates the importance of the content displayed by the teleprompter and dynamically screens it based on the content similarity, so that the most relevant information can be highlighted and displayed preferentially according to the content that the user is concerned about. By adjusting the displayed content in real time, it is ensured that the user can always obtain the key information that he is concerned about, thereby improving the efficiency and practicality of the teleprompter.

[0064] 3. The present invention closely combines the highlighted content with the display progress through voice recognition technology and dynamic monitoring of the display progress, thus realizing content priority display adjustment. By accurately adjusting the display rhythm of the teleprompter, it ensures that the content that the user is most concerned about is displayed at the right time, optimizes the content display process, and significantly improves the convenience and response speed of the teleprompter. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic flow chart of an adaptive feedback control method for a teleprompter according to the present invention;

[0066] Figure 2 The present invention is a schematic structural diagram of an adaptive feedback control system for a teleprompter. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Embodiment 1: Figure 1 As shown, the present invention provides a technical solution, an adaptive feedback control method for a teleprompter, and the adaptive feedback control method comprises the following steps:

[0069] Step S1, collecting the user's facial feature data, and analyzing the user's facial orientation based on the facial feature data, and marking the moment when the user meets the facial orientation condition as the key marking moment;

[0070] The user's facial feature data is collected through an image sensor, and the facial feature data includes the user's eye position, nose position and mouth position. The facial feature data is input into a facial recognition algorithm, and the user's facial orientation is analyzed. Based on the user's facial orientation analysis result, it is determined whether the user is facing the teleprompter display screen. When the user's face is facing the teleprompter display screen, the facial orientation condition is met, and the moment when the facial orientation condition is met is marked as the key marked moment.

[0071] Step S2, collecting the user's eye data at the key marking moment, wherein the user's eye data includes the user's pupil position data and corneal reflection position data, and calculating the user's line of sight deviation angle in combination with the user's eye data;

[0072] Step S2-1, with the ground as a reference, construct a three-dimensional space coordinate system with the geometric center of the teleprompter display screen as the origin, and define the directions as follows: the X-axis is parallel to the ground and extends along the horizontal side of the display screen, with the positive direction being from the center to the right and the negative direction being to the left; the Y-axis is perpendicular to the ground and extends along the vertical side of the display screen, with the positive direction being upward and the negative direction being downward; the Z-axis is perpendicular to the plane of the display screen, with the positive direction being from the center to the direction where the user is located and the negative direction being toward the back of the display screen, and using an infrared sensor to collect the user's eye data at the key marking moment, the user's eye data includes the user's pupil position data and the corneal reflection position data; when the user is at the key marking moment, the user's face faces the teleprompter display screen, and the infrared sensor can completely obtain the user's eye data; at any key marking moment, the origin position of the space coordinate system is P0(x0, y0, z0), the position of the pupil position in the space coordinate system is Ppupil(xk, yk, zk), and the position of the corneal reflection position in the space coordinate system is Pcornea(xc, yc, zc);

[0073] Step S2-2, calculate the user's line of sight deviation angle according to the user's eye data at the key marking moment, the pupil position includes the left eye pupil position and the right eye pupil position, the position of the left eye pupil position in the spatial coordinate system is Ppupil, left (xk, l, yk, l, zk, l), the position of the right eye pupil position in the spatial coordinate system is Ppupil, right (xk, r, yk, r, zk, r); the corneal reflection position includes the left eye corneal reflection position and the right eye corneal reflection position, the left eye corneal reflection position The position in the spatial coordinate system is Pcornea,left(xc,l,yc,l,zc,l), the position of the right eye corneal reflection position in the spatial coordinate system is Pcornea,right(xc,r,yc,r,zc,r), the pupil position and corneal reflection position of the left eye are substituted into the line of sight deviation angle calculation formula to obtain the line of sight deviation angle of the left eye, the pupil position and corneal reflection position of the right eye are substituted into the line of sight deviation angle calculation formula to obtain the line of sight deviation angle of the right eye, and the line of sight deviation angle calculation formula is as follows:

[0074]

[0075] Where Wt represents the user's line of sight deviation angle at the time t of the key mark.

[0076] For example, the left eye pupil position coordinates measured by the infrared sensor are Ppupil, left (1, 2, 3), and the right eye pupil position coordinates are Ppupil, right (-1, -2, -3); the left eye corneal reflection position coordinates are Pcornea, left (2, 3, 4), and the right eye corneal reflection position coordinates are Pcornea, right (-2, -3, -4). According to the line of sight deviation angle calculation formula, it can be calculated as follows:

[0077]

[0078] Step S3, calculating the user's sight focus position at the time of key marking through the user's eye data, judging whether the user's gaze on the teleprompter display screen at the time of key marking is effective according to the sight focus position, storing the user's sight deviation angle and gaze time during effective gaze and constructing a user attention data set;

[0079] Step S3-1, calculating the user's sight focus position at the time of key marking according to the user's eye data, the sight focus position calculation process is as follows:

[0080] Step S3-1-1, the pupil position and corneal reflection position of the left eye are brought into the sight direction vector calculation formula to obtain the sight direction vector of the left eye, and the pupil position and corneal reflection position of the right eye are brought into the sight direction vector calculation formula to obtain the sight direction vector of the right eye, the sight direction vector calculation formula is as follows:

[0081] V=(xc-xk, yc-yk, zc-zk);

[0082] Where V is the user's sight direction vector, specifically, the direction from the pupil position of the eye to the object through the corneal reflection point position;

[0083] Step S3-2-2, calculate the sight focus position according to the user's eye data and the sight direction vector V, the sight focus position is represented as the intersection of the left eye sight direction vector and the right eye sight direction vector, the calculation process is as follows:

[0084] In the line of sight of the left eye, any point Rl(tl) is expressed as:

[0085] Rl(tl)=Ppupil,left+tl*Vl;

[0086] Wherein, Rl(tl) represents a point in the sight direction of the left eye; tl represents a scalar parameter, specifically the distance between point Rl and the left eye pupil position; Vl represents the sight direction vector of the left eye;

[0087] In the line of sight of the right eye, any point Rr(tr) is expressed as:

[0088] Rr(tr)=Ppupil,right+tr*Vr;

[0089] Where Rr(tr) is a point in the direction of the left eye's sight; tr is a scalar parameter, specifically the distance between point Rr and the pupil position of the right eye; Vr is the sight direction vector of the right eye;

[0090] The focus position of the sight is calculated according to the formula of equal intersection coordinates. The calculation formula is as follows:

[0091] Rl(tl)=Rr(tr);

[0092] According to the condition of equal intersection position, three equations are obtained by respectively substituting the x-axis, y-axis and z-axis. The equations are solved to obtain the scalar parameters tl and tr. The values ​​of tl and tr are substituted to obtain the coordinates of the intersection, which is the focus position of the user's sight at the time of key marking. The coordinates are expressed as (xt, yt, zt);

[0093] Step S3-3: Determine whether the user's gaze at the teleprompter display screen at the key marking moment is an effective gaze based on the calculated position of the user's line-of-sight focus. The determination process is as follows:

[0094] Obtain the edge coordinate data of the effective display area of the teleprompter display screen on the x-axis and y-axis. The plane effective display area refers to the area of the teleprompter display screen that the user can effectively gaze at and pay attention to on the plane formed by the x-axis and y-axis. The edge coordinate data includes the maximum value xmax and minimum value xmin of the display screen on the x-axis, and the maximum value ymax and minimum value ymin on the y-axis. Determine whether the line-of-sight focus position is within the plane effective display area based on the edge coordinate data. The determination process is as follows:

[0095] When xmax ≥ xt ≥ xmin and ymax ≥ yt ≥ ymin, it is determined that the user's line-of-sight focus is located within the plane effective display area;

[0096] When the x-axis coordinate of the focus does not satisfy xmax ≥ xt ≥ xmin, or the y-axis coordinate of the focus does not satisfy ymax ≥ yt ≥ ymin, it is determined that the user's line-of-sight focus is not within the plane effective display area;

[0097] Set the focus error threshold Zmax, and judge the vertical distance from the line-of-sight focus position to the teleprompter display screen according to the z-axis coordinate data of the line-of-sight focus position. The determination process is as follows:

[0098] When |zt - z0| ≤ Zmax, it is determined that the line-of-sight focus position meets the condition of effective gaze in terms of vertical distance;

[0099] When |zt - z0| > Zmax, it is determined that the line-of-sight focus position does not meet the condition of effective gaze in terms of vertical distance;

[0100] When the line-of-sight focus position simultaneously meets the conditions of being located within the plane effective display area and meeting the condition of effective gaze in terms of vertical distance, it is determined that the user's gaze at the teleprompter display screen is an effective gaze; when the line-of-sight focus position does not meet the condition of being located within the plane effective display area, or does not meet the condition of effective gaze in terms of vertical distance, it is determined that the user's gaze at the teleprompter display screen is an ineffective gaze;

[0101] Step S3-4: Obtain the fixation time T from the moment when the user's line-of-sight focus first enters the effective display area to the moment when the user's line-of-sight focus leaves this area, and store the user's line-of-sight deviation angle and fixation time to construct a user attention data set.

[0102] For example, the left eye pupil position coordinates obtained by the infrared sensor are Ppupil, left (0, 10, 5), and the right eye pupil position coordinates are Ppupil, right (0.06, 10, 5); the left eye corneal reflection position coordinates are Pcornea, left (0, 10.1, 5.1), and the right eye corneal reflection position coordinates are Pcornea, right (0.06, 10.1, 5.1). According to the line of sight direction vector calculation formula, we can get:

[0103] Vleft=(0-0, 10.1-10, 5.1-5)=(0, 0.1, 0.1);

[0104] Vright=(0.06-0.06, 10.1-10.5, 5.1-5)=(0, 0.1, 0.1);

[0105] Assume tl = tr = 1, that is, the two lines of sight meet at the same distance, and then bring the line of sight focus into the calculation to get:

[0106] xt=0.06 / 2=0.03;

[0107] yt=10+0.1*(tl+tr) / 2=10+0.1*1=10.1;

[0108] zt=5+0.1*(tl+tr) / 2=5+0.1*1=5.1;

[0109] The visual focus position is obtained as (0.03, 10.1, 5.1);

[0110] In the specific implementation process, a two-dimensional coordinate is obtained by obtaining the x-axis and y-axis coordinates of the focus. Then, based on the x-axis and y-axis coordinates of the teleprompter display screen, its plane effective display area is limited. Further, based on the distance perpendicular to the teleprompter display screen, it is determined whether the user's line of sight falls within the effective area of ​​the display screen. This process ensures the matching relationship between the focus of the user's line of sight and the teleprompter display screen.

[0111] Step S4, setting a content similarity threshold, calculating the importance of the content currently displayed on the teleprompter display screen according to the user attention data set, analyzing and screening the undisplayed content in combination with the content similarity threshold, and highlighting the analyzed and screened content;

[0112] Step S4-1, set the content importance threshold Imin and the content similarity threshold Umin, and calculate the importance of the content currently displayed on the teleprompter display screen through the user attention data set. The importance calculation formula is as follows:

[0113]

[0114] Where, Icontent represents the importance of the content of effective gaze; ET represents the weighting coefficient of gaze time; T represents the gaze time; EW represents the weighting coefficient of the sight deviation angle; Wmax represents the maximum value of the sight deviation angle, specifically the maximum deviation angle received by the teleprompter display screen;

[0115] Step S4-2, filter out the content with Icontent≥Imin, and use the filtered content as reference content, input the undisplayed content and the reference content into the neural network to calculate the content similarity, and highlight the undisplayed content with content similarity greater than Umin according to the calculation result.

[0116] For example, the sight deviation angle W = 31.17°, the fixation time T = 3 seconds, the maximum sight deviation angle Wmax = 104°, the weighting coefficient of fixation time ET = 0.6, the weighting coefficient of sight deviation angle EW = 0.4, the content importance threshold Imin = 2, the content similarity threshold Umin = 0.8, according to the content importance calculation formula:

[0117] Icontent=0.6*3*0.4*(104 / 31.17)≈2.4;

[0118] Step S5, using voice recognition technology to monitor the content display progress of the teleprompter display screen in real time, and adjusting the highlighted content to be displayed with priority according to the content display progress. The priority display adjustment means that during the content display process of the teleprompter, the highlighted content is adjusted to be displayed with priority.

[0119] Set the content progress threshold Lmax, monitor the content display progress Lcurrent of the teleprompter display screen in real time through voice recognition technology, and identify the content that the teleprompter needs to display next, and judge whether it is the highlighted content. When the content that needs to be displayed next is the highlighted content, judge the progress of the currently displayed content. The judgment process is as follows:

[0120] When Lcurrent ≥ Lmax, adjust the content of the teleprompter display screen to the content highlighted in the next step;

[0121] When Lcurrent<Lmax, the contents of the teleprompter display screen continue to be displayed.

[0122] During the specific implementation process, the calculated content importance value is compared with the content importance threshold Imin, and the content that is greater than the content importance threshold is highlighted. When the content to be displayed in the next step is read as highlighted content, the content display progress of the teleprompter display screen is monitored in real time through voice recognition technology. When the progress meets the priority display adjustment, the highlighted content to be displayed in the next step will be displayed in advance.

[0123] Embodiment 2, as Figure 2 As shown, the present invention provides an adaptive feedback control system for a teleprompter, the adaptive feedback control system comprising a facial analysis module, a sight line calculation module, a focus judgment module, a content analysis module and a display control module;

[0124] The facial analysis module is used to analyze whether the user is facing the teleprompter through an image sensor and mark key moments; the sightline calculation module is used to calculate the sightline deviation angle based on the user's eye data to determine whether the user's sight is deviating from the teleprompter display screen; the focus judgment module is used to determine whether the user's sight is concentrated in the effective display area and whether the gaze is effective; the content analysis module is used to calculate the importance of the content that the user is paying attention to and highlight it according to the similarity; the display control module is used to adjust the display content according to the progress of voice recognition to ensure that the highlighted content is displayed in advance;

[0125] The output end of the facial analysis module is electrically connected to the input end of the sight line calculation module; the output end of the sight line calculation module is electrically connected to the input end of the focus judgment module; the output end of the focus judgment module is electrically connected to the input end of the content analysis module, and the output end of the content analysis module is electrically connected to the input end of the display control module;

[0126] The facial analysis module includes an image acquisition unit and a direction analysis unit; the image acquisition unit is used to acquire user facial image data; the direction analysis unit is used to analyze whether the user is facing the teleprompter;

[0127] The sight line calculation module includes an infrared acquisition unit and a deviation calculation unit; the infrared acquisition unit is used to collect user eye data; the deviation calculation unit is used to calculate the deviation angle of the sight line;

[0128] The focus judgment module includes a focus calculation unit and a gaze judgment unit; the focus calculation unit is used to calculate the user's line of sight focus position; the gaze judgment unit is used to judge whether the user's gaze is valid;

[0129] The content analysis module includes an importance calculation unit and a similarity analysis unit; the importance calculation unit is used to calculate the importance of the content; the similarity analysis unit is used to analyze the similarity between the undisplayed content and the reference content;

[0130] The display control module includes a progress monitoring unit and a priority display unit; the progress monitoring unit is used to monitor the content display progress of the teleprompter in real time; the priority display unit is used to adjust the highlighted content according to the progress.

[0131] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An adaptive feedback control method for a teleprompter, characterized in that: The adaptive feedback control method comprises the following steps: Step S1, collecting the user's facial feature data, and analyzing the user's facial orientation based on the facial feature data, and marking the moment when the user meets the facial orientation condition as the key marking moment; Step S2, collecting the user's eye data at the key marking moment, wherein the user's eye data includes the user's pupil position data and corneal reflection position data, and calculating the user's line of sight deviation angle in combination with the user's eye data; Step S3, calculating the user's sight focus position at the time of key marking through the user's eye data, judging whether the user's gaze on the teleprompter display screen at the time of key marking is effective according to the sight focus position, storing the user's sight deviation angle and gaze time during effective gaze and constructing a user attention data set; Step S4, setting a content similarity threshold, calculating the importance of the content currently displayed on the teleprompter display screen according to the user attention data set, analyzing and screening the undisplayed content in combination with the content similarity threshold, and highlighting the analyzed and screened content; Step S5, using voice recognition technology to monitor the content display progress of the teleprompter display screen in real time, and adjusting the highlighted content to be displayed with priority according to the content display progress. The priority display adjustment means that during the content display process of the teleprompter, the highlighted content is adjusted to be displayed with priority.

2. The method for adaptive feedback control of a teleprompter according to claim 1, characterized in that: In step S1, facial feature data of the user is collected by an image sensor, wherein the facial feature data includes the eye position, nose position and mouth position of the user, and the facial feature data is input into a facial recognition algorithm to analyze the facial orientation of the user. Based on the facial orientation analysis result, it is determined whether the user is facing the teleprompter display screen. When the user's face is facing the teleprompter display screen, the facial orientation condition is met, and the moment when the facial orientation condition is met is marked as the marked moment.

3. The adaptive feedback control method for a teleprompter according to claim 2, characterized in that: The specific steps of step S2 are as follows: Step S2-1, with the ground as the reference, construct a three-dimensional space coordinate system with the geometric center of the teleprompter display screen as the origin, and define the directions as follows: the X-axis is parallel to the ground and extends along the horizontal side of the display screen, with the positive direction being from the center to the right and the negative direction being to the left; the Y-axis is perpendicular to the ground and extends along the vertical side of the display screen, with the positive direction being upward and the negative direction being downward; The Z axis is perpendicular to the plane of the display screen, with the positive direction extending from the center to the direction where the user is, and the negative direction pointing to the back of the display screen. The infrared sensor is used to collect the user's eye data at the key marking moment, and the user's eye data includes the user's pupil position data and corneal reflection position data; When the user is marking a key moment, the user faces the teleprompter display screen, and the infrared sensor can fully obtain the user's eye data; At any key marking moment, the origin of the spatial coordinate system is P0 (x0, y0, z0), the pupil position in the spatial coordinate system is Ppupil (xk, yk, zk), and the corneal reflection position in the spatial coordinate system is Pcornea (xc, yc, zc); Step S2-2, calculate the user's line of sight deviation angle according to the user's eye data at the key marking moment, the pupil position includes the left eye pupil position and the right eye pupil position, the position of the left eye pupil position in the spatial coordinate system is Ppupil, left (xk, l, yk, l, zk, l), the position of the right eye pupil position in the spatial coordinate system is Ppupil, right (xk, r, yk, r, zk, r); the corneal reflection position includes the left eye corneal reflection position and the right eye corneal reflection position, the left eye corneal reflection position The position in the spatial coordinate system is Pcornea,left(xc,l,yc,l,zc,l), the position of the right eye corneal reflection position in the spatial coordinate system is Pcornea,right(xc,r,yc,r,zc,r), the pupil position and corneal reflection position of the left eye are substituted into the line of sight deviation angle calculation formula to obtain the line of sight deviation angle of the left eye, the pupil position and corneal reflection position of the right eye are substituted into the line of sight deviation angle calculation formula to obtain the line of sight deviation angle of the right eye, and the line of sight deviation angle calculation formula is as follows: Where Wt represents the user's line of sight deviation angle at the time t of the key mark.

4. The adaptive feedback control method for a teleprompter according to claim 3, characterized in that: The specific steps of step S3 are as follows: Step S3-1, calculating the user's sight focus position at the time of key marking according to the user's eye data, the sight focus position calculation process is as follows: Step S3-1-1, the pupil position and corneal reflection position of the left eye are brought into the sight direction vector calculation formula to obtain the sight direction vector of the left eye, and the pupil position and corneal reflection position of the right eye are brought into the sight direction vector calculation formula to obtain the sight direction vector of the right eye, the sight direction vector calculation formula is as follows: V=(xc-xk, yc-yk, zc-zk); Where V is the user's sight direction vector, specifically, the direction from the pupil position of the eye to the object through the corneal reflection point position; Step S3-2-2, calculate the sight focus position according to the user's eye data and the sight direction vector V, the sight focus position is represented as the intersection of the left eye sight direction vector and the right eye sight direction vector, the calculation process is as follows: In the line of sight of the left eye, any point Rl(tl) is expressed as: Rl(tl)=Ppupil,left+tl*Vl; Wherein, Rl(tl) represents a point in the sight direction of the left eye; tl represents a scalar parameter, specifically the distance between point Rl and the pupil position of the left eye; Vl represents the sight direction vector of the left eye; In the direction of the right eye’s line of sight, any point Rr(tr) is expressed as: Rr(tr)=Ppupil,right+tr*Vr; Where Rr(tr) is a point in the direction of the left eye's sight; tr is a scalar parameter, specifically the distance between point Rr and the pupil position of the right eye; Vr is the sight direction vector of the right eye; The focus position of the sight is calculated according to the formula of equal intersection coordinates. The calculation formula is as follows: Rl(tl)=Rr(tr); According to the condition of equal intersection point positions, three equations are obtained by substituting into the x-axis, y-axis, and z-axis respectively. Solving the equations gives the scalar parameters tl and tr, and substituting the values of tl and tr to obtain the intersection point coordinates, which are the line-of-sight focus positions of the user at the key marking moment, and the coordinates are expressed as (xt, yt, zt).

5. The method for adaptive feedback control of a teleprompter according to claim 4, characterized in that: In step S3, it further includes: Step S3-3: Determine whether the user's gaze at the teleprompter display screen at the key marking moment is a valid gaze according to the calculated line-of-sight focus position of the user. The determination process is as follows: Obtain the edge coordinate data of the plane effective display area of the teleprompter display screen on the x-axis and y-axis. The plane effective display area is the area on the plane formed by the x-axis and y-axis where the user can effectively gaze and pay attention to the display of the teleprompter display screen. The edge coordinate data includes the maximum value xmax and minimum value xmin of the display screen on the x-axis, and the maximum value ymax and minimum value ymin on the y-axis. Determine whether the line-of-sight focus position is in the plane effective display area according to the edge coordinate data. The determination process is as follows: When xmax≥xt≥xmin and ymax≥yt≥ymin, it is determined that the user's line-of-sight focus is located in the plane effective display area; When the x-axis coordinate of the focus does not satisfy xmax≥xt≥xmin, or the y-axis coordinate of the focus does not satisfy ymax≥yt≥ymin, it is determined that the user's line-of-sight focus is not in the plane effective display area; Set the focus error threshold Zmax, and judge the vertical distance from the line-of-sight focus position to the teleprompter display screen according to the z-axis coordinate data of the line-of-sight focus position. The judgment process is as follows: When 丨zt - z0丨≤Zmax, it is determined that the line-of-sight focus position satisfies valid gaze in the vertical distance; When 丨zt - z0丨>Zmax, it is determined that the line-of-sight focus position does not satisfy valid gaze in the vertical distance; When the line-of-sight focus position simultaneously satisfies being located in the plane effective display area and satisfying the condition of valid gaze in the vertical distance, it is determined that the user's gaze at the teleprompter display screen is a valid gaze; when the line-of-sight focus position does not satisfy being located in the plane effective display area, or does not satisfy the condition of valid gaze in the vertical distance, it is determined that the user's gaze at the teleprompter display screen is an invalid gaze; Step S3-4: Obtain the fixation time T from the moment when the user's line-of-sight focus first enters the effective display area to the moment when the user's line-of-sight focus leaves the area, and store the user's line-of-sight deviation angle and fixation time to construct a user attention data set.

6. The method for adaptive feedback control of a teleprompter according to claim 5, characterized in that: The specific steps of step S4 are as follows: Step S4-1: Set the content importance threshold Imin and the content similarity threshold Umin, and calculate the importance of the current display content of the teleprompter display screen through the user attention data set. The importance calculation formula is as follows: In the formula, Icontent represents the importance of the content of the valid gaze; ET represents the weighting coefficient of the fixation time; T represents the fixation time; EW represents the weighting coefficient of the line-of-sight deviation angle; Wmax represents the maximum value of the line-of-sight deviation angle, specifically the maximum deviation angle received by the teleprompter display screen; Step S4-2, filter out the content with Icontent≥Imin, and use the filtered content as reference content, input the undisplayed content and the reference content into the neural network to calculate the content similarity, and highlight the undisplayed content with content similarity greater than Umin according to the calculation result.

7. The method for adaptive feedback control of a teleprompter according to claim 6, characterized in that: In step S5, the content progress threshold Lmax is set, and the content display progress Lcurrent of the teleprompter display screen is monitored in real time through voice recognition technology, and the content that the teleprompter needs to display next is identified to determine whether it is the highlighted content. When the content that needs to be displayed next is the highlighted content, the progress of the currently displayed content is determined. The determination process is as follows: When Lcurrent ≥ Lmax, adjust the content of the teleprompter display screen to the content highlighted in the next step; When Lcurrent<Lmax, the contents of the teleprompter display screen continue to be displayed.

8. An adaptive feedback control system for a teleprompter, applied to an adaptive feedback control method for a teleprompter according to any one of claims 1 to 7, characterized in that: The adaptive feedback control system includes a facial analysis module, a sight line calculation module, a focus judgment module, a content analysis module and a display control module; The facial analysis module is used to analyze whether the user is facing the teleprompter through an image sensor and mark key moments; the sightline calculation module is used to calculate the sightline deviation angle based on the user's eye data and determine whether the user's sight is deviating from the teleprompter display screen; the focus judgment module is used to determine whether the user's sight is concentrated in the effective display area and determine whether the gaze is effective; the content analysis module is used to calculate the importance of the content that the user is paying attention to and highlight it according to the similarity; The display control module is used to adjust the display content according to the progress of speech recognition to ensure that the highlighted content is displayed in advance; The output end of the facial analysis module is electrically connected to the input end of the sight line calculation module; the output end of the sight line calculation module is electrically connected to the input end of the focus judgment module; the output end of the focus judgment module is electrically connected to the input end of the content analysis module, and the output end of the content analysis module is electrically connected to the input end of the display control module.

9. The adaptive feedback control system for a teleprompter according to claim 8, characterized in that: The facial analysis module includes an image acquisition unit and a direction analysis unit; the image acquisition unit is used to acquire user facial image data; the direction analysis unit is used to analyze whether the user is facing the teleprompter; The sight line calculation module includes an infrared acquisition unit and a deviation calculation unit; the infrared acquisition unit is used to collect user eye data; The deviation calculation unit is used to calculate the deviation angle of the line of sight; The focus judgment module includes a focus calculation unit and a gaze judgment unit; the focus calculation unit is used to calculate the user's line of sight focus position; The gaze determination unit is used to determine whether the user's gaze is valid.

10. The adaptive feedback control system for a teleprompter according to claim 8, characterized in that: The content analysis module includes an importance calculation unit and a similarity analysis unit; the importance calculation unit is used to calculate the importance of the content; the similarity analysis unit is used to analyze the similarity between the undisplayed content and the reference content; The display control module includes a progress monitoring unit and a priority display unit; the progress monitoring unit is used to monitor the content display progress of the teleprompter in real time; the priority display unit is used to adjust the highlighted content according to the progress.

Images