Blind operation assisting method and system based on smart recruitment platform of gesture interaction

Abstract

The application relates to the technical field of gesture interaction, and provides a blind operation auxiliary method and system of an intelligent recruitment platform based on gesture interaction, which comprises the following steps: collecting identity information and calling corresponding interface element distribution configuration data; generating virtual focus positioning information according to the interface element distribution configuration data, mapping the virtual focus positioning information into a gesture recognition coordinate system, and obtaining interface-gesture correlation information; performing trajectory form recognition on dynamic gesture trajectory data to obtain a gesture intention feature vector; matching and comparing the gesture intention feature vector with the virtual focus positioning information to determine a target interface element and a target operation instruction; applying the target operation instruction, obtaining interface feedback information, converting the interface feedback information into a voice broadcast signal, and outputting the voice broadcast signal; and performing effectiveness verification on the confirmation gesture data to obtain a blind operation auxiliary result; and the application can improve the reliability of blind operation auxiliary of the intelligent recruitment platform.

Description

Technical Field

[0001] This invention relates to the field of gesture interaction technology, and in particular to a blind operation assistance method and system for an intelligent enrollment platform based on gesture interaction. Background Technology

[0002] As the functional system of the intelligent enrollment platform continues to improve, the number of business operation interfaces and interactive elements on the platform continues to grow, placing higher demands on the ease of operation in scenarios without visual assistance. Existing enrollment platforms generally rely on visual guidance, depending on the user's visual positioning and precise touch control of interface elements to complete operations, which cannot support the normal usage needs of visually impaired individuals or those operating blindly. Conventional gesture interaction solutions often use generalized trajectory recognition logic without establishing a dedicated mapping system for the distribution of interface elements on the enrollment platform. The conversion accuracy between the gesture recognition coordinate system and the interface coordinate system is insufficient, and the matching deviation between virtual focus and actual interactive elements is significant, directly resulting in low accuracy in locating target elements under blind operation and difficulty in guaranteeing operational efficiency.

[0003] Existing blind operation assistance technologies have significant shortcomings in feedback mechanisms and operation verification. Most solutions can only output simple voice information of operation results, without simultaneously integrating information about surrounding elements and layout prompts at the current focus position. Users cannot establish spatial awareness of the interface through voice feedback, easily leading to lost operation paths and invalid repetitive operations. Furthermore, existing solutions generally lack a complete mechanism for verifying the validity of confirmation gestures, failing to comprehensively verify user confirmation commands from multiple dimensions such as trajectory direction, amplitude, outline, and duration. This results in a high rate of false triggering. Moreover, they do not incorporate personalized interface configurations based on user identity characteristics and historical operating habits, making it difficult to adapt to the gesture operation characteristics of different users. Overall, the reliability and user experience of blind operation assistance are significantly inadequate. Therefore, improving the reliability of blind operation assistance in intelligent enrollment platforms has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a blind operation assistance method and system for an intelligent enrollment platform based on gesture interaction, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a blind operation assistance method for an intelligent enrollment platform based on gesture interaction, comprising:

[0006] A. In response to the user's entry into the intelligent enrollment platform, collect the user's identity information and retrieve the interface element distribution configuration data corresponding to the user based on the identity information;

[0007] B. Based on the interface element distribution configuration data, generate virtual focus positioning information corresponding to the interactive elements in the intelligent enrollment platform, and map the virtual focus positioning information to the gesture recognition coordinate system of the intelligent enrollment platform to obtain the interface-gesture association information of the intelligent enrollment platform;

[0008] C. Collect the user's dynamic gesture trajectory data in real time, perform trajectory shape recognition on the dynamic gesture trajectory data, and obtain the user's gesture intention feature vector;

[0009] D. Based on the interface-gesture association information, the gesture intention feature vector is matched and compared with the virtual focus positioning information, and the user's target interface element and target operation command are determined according to the comparison result;

[0010] E. Apply the target operation instruction to obtain the interface feedback information of the target interface element, and convert the interface feedback information into a voice broadcast signal, wherein the voice broadcast signal includes operation result description information and current focus position prompt information;

[0011] F. Output the voice broadcast signal, collect the user's confirmation gesture data, and verify the validity of the confirmation gesture data to obtain the user's blind operation assistance result.

[0012] In a preferred embodiment, the step of responding to the user's access to the intelligent enrollment platform by collecting the user's identity information and retrieving the interface element distribution configuration data corresponding to the user based on the identity information includes:

[0013] The system receives an entry request signal triggered when a user enters the intelligent enrollment platform, and extracts the user's identity information from the entry request signal. The identity information includes the user's login account information and device fingerprint information.

[0014] Based on the login account information, the user configuration database of the intelligent enrollment platform is retrieved to obtain the user's interface configuration record. The interface configuration record includes the user's custom interface layout parameters and historical operation preference tags.

[0015] Based on the device fingerprint information, the interface configuration records are matched and filtered to obtain the user's interface element distribution configuration data.

[0016] In a preferred embodiment, generating virtual focus positioning information corresponding to interactive elements in the intelligent enrollment platform based on the interface element distribution configuration data includes:

[0017] The interface position parameters and interface size parameters of the interactive elements are extracted from the interface element distribution configuration data. The interface position parameters include the horizontal and vertical coordinate values ​​of the interactive element in the operation interface of the intelligent enrollment platform. The interface size parameters include the width and height values ​​of the interactive element.

[0018] The interface position parameters and interface size parameters are processed to obtain the virtual focus candidate positions of the interactive elements by performing center point positioning processing.

[0019] Assign a unique virtual focus identifier to the interactive element, and associate the virtual focus identifier with the virtual focus candidate position to obtain the virtual focus positioning record of the interactive element;

[0020] Based on the interface hierarchy order in the interface element distribution configuration data, the virtual focus positioning records are arranged to obtain the virtual focus positioning information of the intelligent enrollment platform.

[0021] In a preferred embodiment, mapping the virtual focus positioning information to the gesture recognition coordinate system of the intelligent enrollment platform to obtain the interface-gesture association information of the intelligent enrollment platform includes:

[0022] Using the top left corner of the intelligent enrollment platform as the origin, with the horizontal direction to the right as the x-axis and the vertical direction downward as the y-axis, a gesture recognition coordinate system for the intelligent enrollment platform is constructed.

[0023] Based on the coordinate origin position and coordinate axis direction definition of the gesture recognition coordinate system, the virtual focus candidate position is transformed from the interface coordinate system to the gesture recognition coordinate system to obtain the focus coordinate value of the interactive element;

[0024] Assign corresponding gesture recognition region boundary parameters to the focus coordinate values;

[0025] The focus coordinates, the gesture recognition area boundary parameters, and the virtual focus identifier are associated and stored to obtain the interface-gesture association information of the intelligent enrollment platform.

[0026] In a preferred embodiment, the real-time acquisition of the user's dynamic gesture trajectory data, and the performance of trajectory shape recognition on the dynamic gesture trajectory data to obtain the user's gesture intent feature vector, includes:

[0027] Acquire the user's dynamic gesture trajectory data, which includes a sequence of coordinate values ​​for trajectory sampling points;

[0028] The coordinate value sequence is cleaned, and the cleaned coordinate value sequence is interpolated and filled to obtain the user's effective trajectory point sequence;

[0029] Perform directional change analysis on adjacent trajectory sampling points in the effective trajectory point sequence to obtain the user's trajectory direction change sequence;

[0030] The distance between the first and last sampling points in the effective trajectory point sequence is taken as the trajectory amplitude value of the effective trajectory point sequence;

[0031] Contour recognition is performed on the effective trajectory point sequence to obtain the trajectory contour type of the effective trajectory point sequence, and the trajectory contour type includes straight line, arc and polyline;

[0032] The user's gesture intent feature vector is obtained by fusing the trajectory direction change sequence, the trajectory amplitude value, and the trajectory contour type.

[0033] In a preferred embodiment, the step of matching and comparing the gesture intent feature vector with the virtual focus positioning information based on the interface-gesture association information, and determining the user's target interface element and target operation command based on the comparison result, includes:

[0034] Extract the focus coordinates and gesture recognition area boundary parameters of the interactive elements from the interface-gesture association information, and extract the trajectory endpoint coordinates from the gesture intent feature vector.

[0035] Distance analysis is performed on the focal coordinates and the trajectory endpoint coordinates to obtain the spatial distance between the focal coordinates and the trajectory endpoint coordinates;

[0036] Based on the spatial distance value and the gesture recognition area boundary parameters, the matching degree value of the interactive element is determined, and the interactive element with the highest matching degree value is selected as the user's target interface element.

[0037] The element type attribute of the target interface element and the trajectory contour type in the gesture intention feature vector are encoded into the user's target operation instruction.

[0038] In a preferred embodiment, the step of applying the target operation instruction, obtaining interface feedback information of the target interface element, and converting the interface feedback information into a voice broadcast signal, wherein the voice broadcast signal includes operation result description information and current focus position prompt information, including:

[0039] Submit the target operation instruction to the operation execution interface of the intelligent enrollment platform, and receive the execution result data returned by the operation execution interface;

[0040] Information is extracted from the execution result data to obtain the interface feedback information of the target interface element. The interface feedback information includes the operation status identifier, the result data payload, and the coordinates of the current interface focus position.

[0041] The interface feedback information is parsed to obtain the execution status category and feedback content type of the interface feedback information;

[0042] Based on the execution status category and the feedback content type, a template matching search is performed on the preset broadcast template library to obtain the broadcast content template of the target interface element, and the specific data content in the result data payload is filled into the broadcast content template to obtain the operation result description information of the target interface element;

[0043] Extract the focus coordinate value corresponding to the target interface element from the interface-gesture association information to obtain the focus position coordinates of the target interface element;

[0044] Based on the focus position coordinates, the interface-gesture association information is traversed to obtain the focus dwelling element of the target interface element, and the element name and element function description of the focus dwelling element are combined to form the current focus position prompt information of the target interface element.

[0045] The operation result description information and the current focus position prompt information are concatenated into a broadcast text sequence of the target interface element, and the speech synthesis parameters of the broadcast text sequence are configured to obtain the speech broadcast signal of the intelligent enrollment platform.

[0046] In a preferred embodiment, the step of validating the confirmation gesture data to obtain the user's blind operation assistance result includes:

[0047] The user's confirmation gesture data is obtained, and the confirmation gesture data is analyzed in multiple dimensions to obtain the confirmation trajectory amplitude value, confirmation trajectory direction change sequence, confirmation trajectory outline type, and confirmation gesture duration.

[0048] The starting direction vector and ending direction vector of the confirmation gesture data are determined based on the confirmed trajectory direction change sequence, and the trajectory turning angle of the confirmation gesture data is determined based on the starting direction vector and the ending direction vector.

[0049] The sequence of confirmed trajectory direction changes is compared item by item with the reference direction vector sequence of the target interface element to obtain the direction matching degree value of the confirmed gesture data;

[0050] The amplitude value of the confirmed trajectory is compared and analyzed with the amplitude reference value of the target interface element to obtain the amplitude matching degree value of the confirmed gesture data;

[0051] The matching degree of the confirmed trajectory contour type and the preset confirmed gesture contour type is measured to obtain the contour matching degree value of the confirmed gesture data, and the duration matching degree value between the duration of the confirmed gesture and the preset standard duration range is calculated.

[0052] The comprehensive verification value of the confirmation gesture data is calculated based on the direction matching value, the amplitude matching value, the contour matching value, and the duration matching value.

[0053] Based on the comprehensive verification value, the operation execution result corresponding to the target operation instruction is used as the blind operation assistance result for the user.

[0054] In a preferred embodiment, the formula for calculating the comprehensive verification value is:

[0055] ;

[0056] in, This represents the comprehensive verification value. This indicates the direction matching degree value. This indicates the amplitude matching degree value. This represents the contour matching degree value. This indicates the duration matching degree value. This indicates the preset duration adjustment coefficient.

[0057] To address the aforementioned problems, this invention also provides a blind operation assistance system for an intelligent enrollment platform based on gesture interaction, the system comprising:

[0058] The information collection module is used to respond to the user's entry into the intelligent enrollment platform, collect the user's identity information, and retrieve the interface element distribution configuration data corresponding to the user based on the identity information.

[0059] The interface-gesture association module is used to generate virtual focus positioning information corresponding to interactive elements in the intelligent enrollment platform based on the interface element distribution configuration data, and to map the virtual focus positioning information to the gesture recognition coordinate system of the intelligent enrollment platform to obtain the interface-gesture association information of the intelligent enrollment platform.

[0060] The trajectory recognition module is used to collect the user's dynamic gesture trajectory data in real time, perform trajectory shape recognition on the dynamic gesture trajectory data, and obtain the user's gesture intention feature vector.

[0061] The information matching module is used to match and compare the gesture intention feature vector with the virtual focus positioning information based on the interface-gesture association information, and determine the user's target interface element and target operation instruction based on the comparison result.

[0062] The voice broadcast module is used to apply the target operation command, obtain the interface feedback information of the target interface element, and convert the interface feedback information into a voice broadcast signal. The voice broadcast signal includes operation result description information and current focus position prompt information.

[0063] The operation assistance module is used to output the voice broadcast signal, collect the user's confirmation gesture data, and verify the validity of the confirmation gesture data to obtain the user's blind operation assistance result.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. This invention retrieves personalized interface element distribution configurations through identity matching, constructs a mapping relationship between virtual focus positioning information and gesture recognition coordinate system, and combines dynamic gesture trajectory morphology recognition and intent feature extraction to accurately locate target interface elements and parse corresponding operation instructions, effectively improving the interactive response speed and operation matching accuracy of blind operation assistance in the intelligent enrollment platform.

[0066] 2. This invention transforms interface feedback information into a voice broadcast signal that includes a description of the operation result and a current focus prompt, and verifies the validity of gestures through multi-dimensional parameter verification, forming a complete blind operation execution closed loop. This enhances the perceptibility and reliability of the operation process, and comprehensively improves the user experience and operational stability of the blind operation assistance of the intelligent enrollment platform. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating a blind operation assistance method for an intelligent enrollment platform based on gesture interaction, provided in an embodiment of the present invention.

[0068] Figure 2 This is a functional module diagram of a blind operation assistance system for an intelligent enrollment platform based on gesture interaction provided in an embodiment of the present invention;

[0069] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0070] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0071] This application provides a blind operation assistance method for an intelligent enrollment platform based on gesture interaction. The execution subject of this gesture-based intelligent enrollment platform blind operation assistance method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application embodiment: a server, a terminal, etc. In other words, the gesture-based intelligent enrollment platform blind operation assistance method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0072] Reference Figure 1 The diagram shown is a flowchart illustrating a blind operation assistance method for an intelligent enrollment platform based on gesture interaction, according to an embodiment of the present invention. In this embodiment, the blind operation assistance method for the intelligent enrollment platform based on gesture interaction includes:

[0073] A. In response to the user's entry into the intelligent enrollment platform, collect the user's identity information and retrieve the interface element distribution configuration data corresponding to the user based on the identity information;

[0074] In this embodiment of the invention, the step of responding to the trigger action of a user entering the intelligent enrollment platform, collecting the user's identity information, and retrieving the interface element distribution configuration data corresponding to the user based on the identity information includes:

[0075] The system receives an entry request signal triggered when a user enters the intelligent enrollment platform, and extracts the user's identity information from the entry request signal. The identity information includes the user's login account information and device fingerprint information.

[0076] Based on the login account information, the user configuration database of the intelligent enrollment platform is retrieved to obtain the user's interface configuration record. The interface configuration record includes the user's custom interface layout parameters and historical operation preference tags.

[0077] Based on the device fingerprint information, the interface configuration records are matched and filtered to obtain the user's interface element distribution configuration data.

[0078] After a user triggers the operation to enter the intelligent enrollment platform, the platform front-end generates an entry request signal and transmits it to the platform's service processing node. The service processing node performs message decomposition and field parsing on the entry request signal, extracts the user's login account information from the user authentication field in the signal, and extracts the device fingerprint information composed of the device hardware serial number, system version identifier, and application installation identifier from the device environment field in the signal. The login account information and device fingerprint information are then integrated into the user's identity information.

[0079] Using the extracted login account information as the retrieval basis, a configuration retrieval request is initiated to the user configuration database of the intelligent enrollment platform. The user configuration database traverses all user configuration entries stored within itself, matches the configuration entry that is completely consistent with the login account information, extracts the custom interface layout parameters and historical operation preference tags contained in the configuration entry, and integrates the two to form the corresponding user's interface configuration record.

[0080] The extracted device fingerprint information is compared one by one with multiple sets of device adaptation configuration items stored in the interface configuration record. A set of device adaptation configuration items that completely matches the current device fingerprint information is selected. This set of device adaptation configuration items, together with the custom interface layout parameters and historical operation preference tags in the interface configuration record, are integrated to finally generate the user's interface element distribution configuration data.

[0081] The beneficial effects are that by verifying the identity through login account information and device fingerprint information, the distribution configuration data of interface elements that match the user's current device can be accurately retrieved, ensuring the device compatibility of subsequent gesture mapping and interactive operations. At the same time, relying on the targeted retrieval and filtering of the user configuration database, the retrieval efficiency and matching accuracy of interface configuration data are improved.

[0082] B. Based on the interface element distribution configuration data, generate virtual focus positioning information corresponding to the interactive elements in the intelligent enrollment platform, and map the virtual focus positioning information to the gesture recognition coordinate system of the intelligent enrollment platform to obtain the interface-gesture association information of the intelligent enrollment platform;

[0083] In this embodiment of the invention, generating virtual focus positioning information corresponding to interactive elements in the intelligent enrollment platform based on the interface element distribution configuration data includes:

[0084] The interface position parameters and interface size parameters of the interactive elements are extracted from the interface element distribution configuration data. The interface position parameters include the horizontal and vertical coordinate values ​​of the interactive element in the operation interface of the intelligent enrollment platform. The interface size parameters include the width and height values ​​of the interactive element.

[0085] The interface position parameters and interface size parameters are processed to obtain the virtual focus candidate positions of the interactive elements by performing center point positioning processing.

[0086] Assign a unique virtual focus identifier to the interactive element, and associate the virtual focus identifier with the virtual focus candidate position to obtain the virtual focus positioning record of the interactive element;

[0087] Based on the interface hierarchy order in the interface element distribution configuration data, the virtual focus positioning records are arranged to obtain the virtual focus positioning information of the intelligent enrollment platform.

[0088] The step of mapping the virtual focus positioning information to the gesture recognition coordinate system of the intelligent enrollment platform to obtain the interface-gesture association information of the intelligent enrollment platform includes:

[0089] Using the top left corner of the intelligent enrollment platform as the origin, with the horizontal direction to the right as the x-axis and the vertical direction downward as the y-axis, a gesture recognition coordinate system for the intelligent enrollment platform is constructed.

[0090] Based on the coordinate origin position and coordinate axis direction definition of the gesture recognition coordinate system, the virtual focus candidate position is transformed from the interface coordinate system to the gesture recognition coordinate system to obtain the focus coordinate value of the interactive element;

[0091] Assign corresponding gesture recognition region boundary parameters to the focus coordinate values;

[0092] The focus coordinates, the gesture recognition area boundary parameters, and the virtual focus identifier are associated and stored to obtain the interface-gesture association information of the intelligent enrollment platform.

[0093] Read the acquired interface element distribution configuration data, traverse all interface element entries with interactive attributes in the data, and extract the interface position parameters and interface size parameters corresponding to each interactive element. The interface position parameters include the horizontal and vertical coordinate values ​​of the interactive element in the operation interface of the intelligent enrollment platform, and the interface size parameters include the width and height values ​​of the interactive element.

[0094] For a single interactive element, the horizontal coordinate value in the element's interface position parameter is used as the starting reference, and half the width value in the element's interface size parameter is added to obtain the horizontal position value of the center point. The vertical coordinate value in the element's interface position parameter is used as the starting reference, and half the height value in the element's interface size parameter is added to obtain the vertical position value of the center point. The horizontal and vertical position values ​​are combined to form the virtual focus candidate position of the interactive element.

[0095] According to the traversal order of interactive elements in the interface, a globally unique virtual focus identifier is assigned to each interactive element. Then, the virtual focus identifier corresponding to each interactive element is bound to its own virtual focus candidate position to form a virtual focus positioning record for each individual interactive element.

[0096] The interface element distribution configuration data is read to determine the interface hierarchy order. All virtual focus positioning records are sorted and integrated according to the hierarchy from high to low, ultimately forming the complete virtual focus positioning information of the intelligent enrollment platform.

[0097] The top left corner of the intelligent enrollment platform's interface is selected as the origin of the coordinate system. The horizontal direction to the right is set as the positive direction of the horizontal axis, and the vertical direction downward is set as the positive direction of the vertical axis. Based on the origin and the direction definitions of the two coordinate axes, the gesture recognition coordinate system of the intelligent enrollment platform is constructed.

[0098] The origin and axis of the gesture recognition coordinate system are defined as the transformation reference. The coordinate values ​​of the virtual focus candidate positions corresponding to each interactive element are converted with reference to the reference of the new coordinate system, so that the coordinate reference of the virtual focus candidate positions is completely consistent with the gesture recognition coordinate system. After the transformation, the focus coordinate values ​​corresponding to each interactive element are obtained.

[0099] Centered on the focus coordinates of each interactive element, a rectangle is used as the default shape of the gesture recognition area. The horizontal width of the recognition area is determined by multiplying the width of the interactive element by 1.2, and the vertical height of the recognition area is determined by multiplying the height of the interactive element by 1.2. The coordinates of the upper left and lower right corners of the rectangle are used as the boundary parameters of the gesture recognition area corresponding to the focus coordinates.

[0100] The focus coordinates, gesture recognition area boundary parameters, and virtual focus identifiers of each interactive element are bound one-to-one and stored centrally to form the interface-gesture association information of the intelligent enrollment platform, which simultaneously includes interface element identifiers, focus coordinates, and gesture recognition range.

[0101] The beneficial effects are that by extracting the position and size parameters of interactive elements and calculating the center point position, accurate virtual focus candidate positions can be generated for each interactive element. Combined with unique identifiers and interface hierarchy sorting, standardized virtual focus positioning information is formed. Then, by constructing a unified gesture recognition coordinate system, coordinate mapping and recognition area delineation are completed, and a stable correspondence between interface elements and gesture recognition space is established. This provides accurate spatial basis for subsequent gesture intent matching, ensuring the accuracy of gesture positioning and the reliability of interaction response under blind operation.

[0102] C. Collect the user's dynamic gesture trajectory data in real time, perform trajectory shape recognition on the dynamic gesture trajectory data, and obtain the user's gesture intention feature vector;

[0103] In this embodiment of the invention, the real-time acquisition of the user's dynamic gesture trajectory data, and the performance of trajectory shape recognition on the dynamic gesture trajectory data to obtain the user's gesture intent feature vector, include:

[0104] Acquire the user's dynamic gesture trajectory data, which includes a sequence of coordinate values ​​for trajectory sampling points;

[0105] The coordinate value sequence is cleaned, and the cleaned coordinate value sequence is interpolated and filled to obtain the user's effective trajectory point sequence;

[0106] Analyze the direction changes of adjacent trajectory sampling points in the effective trajectory point sequence to obtain the user's trajectory direction change sequence;

[0107] The distance between the first and last sampling points in the effective trajectory point sequence is taken as the trajectory amplitude value of the effective trajectory point sequence;

[0108] Contour recognition is performed on the effective trajectory point sequence to obtain the trajectory contour type of the effective trajectory point sequence, and the trajectory contour type includes straight line, arc and polyline;

[0109] The user's gesture intent feature vector is obtained by fusing the trajectory direction change sequence, the trajectory amplitude value, and the trajectory contour type.

[0110] The platform's gesture acquisition device continuously collects the real-time position information of the user's hand at a fixed sampling frequency. The position coordinates of each acquisition are arranged in chronological order according to the acquisition time to form a sequence of coordinate values ​​of trajectory sampling points composed of multiple sets of coordinate values. This sequence of coordinate values ​​is then integrated into the user's dynamic gesture trajectory data.

[0111] The validity of each sampling point in the coordinate value sequence is verified one by one. The verification rules include whether the sampling point position is within the preset collection range of the gesture collection device, and whether the position deviation between the sampling point and the adjacent sampling points is within the preset reasonable range. Abnormal sampling points that do not meet the verification rules are removed to complete data cleaning. Then, a corresponding number of transition position points are added between every two adjacent valid sampling points at a uniform interval to make the sampling points of the entire sequence evenly and continuously distributed, and finally form the user's valid trajectory point sequence.

[0112] In each group of two adjacent trajectory sampling points in the effective trajectory point sequence, the position of the next sampling point is subtracted from the position of the previous sampling point to determine the corresponding motion direction. All the motion directions corresponding to the sampling points are arranged in chronological order according to the acquisition time to obtain the user's trajectory direction change sequence.

[0113] Locate the first sampling point in the effective trajectory point sequence that is collected earlier and the last sampling point that is collected later, calculate the straight-line distance between the two sampling points, and use this straight-line distance as the trajectory amplitude value of the effective trajectory point sequence.

[0114] Traverse all sampling points in the valid trajectory point sequence and calculate the angle of change of motion direction between adjacent sampling points one by one; when the angle of change of direction of all adjacent sampling points is less than 5 degrees, the trajectory contour type is determined to be linear; when the angle of change of direction of all adjacent sampling points is within the range of 5 degrees to 30 degrees and there is no abrupt change of direction exceeding 30 degrees, the trajectory contour type is determined to be arc; when there is at least one adjacent sampling point in the sequence with an angle of change of direction greater than or equal to 30 degrees, the trajectory contour type is determined to be polygonal.

[0115] The generated trajectory direction change sequence, trajectory amplitude value, and trajectory contour type are integrated according to a preset feature combination order. The three types of feature data are then assigned to the corresponding feature terms in the feature vector to form a complete feature vector of the user's gesture intent.

[0116] The beneficial effects are that by collecting users' dynamic gesture trajectories at a fixed frequency and completing standardized data cleaning and interpolation, a continuous and accurate sequence of effective trajectory points can be obtained, providing a reliable data foundation for subsequent trajectory morphology analysis. Then, by analyzing direction changes, calculating amplitude, and recognizing contours, multi-dimensional gesture features are extracted and fused into a gesture intent feature vector, which can comprehensively and accurately represent the user's gesture action intent. This provides accurate feature basis for subsequent interface element matching and operation command generation, effectively improving the recognition accuracy and interaction stability of gesture interaction in blind operation scenarios.

[0117] D. Based on the interface-gesture association information, the gesture intention feature vector is matched and compared with the virtual focus positioning information, and the user's target interface element and target operation command are determined according to the comparison result;

[0118] In this embodiment of the invention, the step of matching and comparing the gesture intent feature vector with the virtual focus positioning information based on the interface-gesture association information, and determining the user's target interface element and target operation command based on the comparison result, includes:

[0119] Extract the focus coordinates and gesture recognition area boundary parameters of the interactive elements from the interface-gesture association information, and extract the trajectory endpoint coordinates from the gesture intent feature vector.

[0120] Distance analysis is performed on the focal coordinates and the trajectory endpoint coordinates to obtain the spatial distance between the focal coordinates and the trajectory endpoint coordinates;

[0121] Based on the spatial distance value and the gesture recognition area boundary parameters, the matching degree value of the interactive element is determined, and the interactive element with the highest matching degree value is selected as the user's target interface element.

[0122] The element type attribute of the target interface element and the trajectory contour type in the gesture intention feature vector are encoded into the user's target operation instruction.

[0123] Retrieve the stored interface-gesture association information, traverse all the associated records corresponding to interactive elements in the information, extract the focus coordinates and gesture recognition area boundary parameters bound to the interactive elements in each record, and at the same time retrieve the generated gesture intent feature vector, extract the trajectory endpoint coordinates corresponding to the end position of the user's dynamic gesture trajectory from the various trajectory feature data items contained in the vector.

[0124] For each interactive element, the positional difference between the focus coordinate and the trajectory endpoint coordinate is compared in the horizontal and vertical directions. The straight line length between the two coordinate points is calculated by combining the positional differences in the two directions, and this straight line length is used as the spatial distance between the focus coordinate and the trajectory endpoint coordinate.

[0125] For each interactive element, the spatial distance value corresponding to the element is compared with the recognition range defined by the boundary parameters of its own gesture recognition area. When the spatial distance value is within the recognition range, the matching degree of the interactive element is determined according to the rule that the smaller the spatial distance value, the higher the matching degree. When the spatial distance value is outside the recognition range, the matching degree of the interactive element is the lowest level. After calculating the matching degree of all interactive elements, the interactive element with the highest matching degree is selected as the user's target interface element.

[0126] Extract the element type attributes inherent to the target interface elements, and simultaneously extract the corresponding trajectory contour type from the gesture intent feature vector. Perform encoding conversion according to the preset encoding mapping rules: the element type attributes are divided into three categories: button type, input box type, and list type, corresponding to codes 01, 02, and 03 respectively; in the trajectory contour type, the straight line type corresponds to the click operation and is encoded as 001, the arc type corresponds to the swipe operation and is encoded as 002, and the polyline type corresponds to the back operation and is encoded as 003; concatenate the operation object code and the operation behavior code in sequence to form a complete target operation instruction.

[0127] The beneficial effect is that by extracting the focus coordinates and gesture recognition area boundary parameters from the interface-gesture association information, and combining the gesture trajectory endpoint coordinates for distance analysis and matching degree calculation, the target interface element pointed to by the user's gesture intention can be accurately located. Then, by combining the type attribute of the target interface element and the gesture trajectory contour type, the operation instruction encoding is completed, realizing the accurate conversion of gesture intention to interface operation, and ensuring the recognition accuracy and operation execution accuracy of gesture interaction in blind operation scenarios.

[0128] E. Apply the target operation instruction to obtain the interface feedback information of the target interface element, and convert the interface feedback information into a voice broadcast signal, wherein the voice broadcast signal includes operation result description information and current focus position prompt information;

[0129] In this embodiment of the invention, the application of the target operation instruction obtains the interface feedback information of the target interface element, and converts the interface feedback information into a voice broadcast signal. The voice broadcast signal includes operation result description information and current focus position prompt information, including:

[0130] Submit the target operation instruction to the operation execution interface of the intelligent enrollment platform, and receive the execution result data returned by the operation execution interface;

[0131] Information is extracted from the execution result data to obtain the interface feedback information of the target interface element. The interface feedback information includes the operation status identifier, the result data payload, and the coordinates of the current interface focus position.

[0132] The interface feedback information is parsed to obtain the execution status category and feedback content type of the interface feedback information;

[0133] Based on the execution status category and the feedback content type, a template matching search is performed on the preset broadcast template library to obtain the broadcast content template of the target interface element, and the specific data content in the result data payload is filled into the broadcast content template to obtain the operation result description information of the target interface element;

[0134] Extract the focus coordinate value corresponding to the target interface element from the interface-gesture association information to obtain the focus position coordinates of the target interface element;

[0135] Based on the focus position coordinates, the interface-gesture association information is traversed to obtain the focus dwelling element of the target interface element, and the element name and element function description of the focus dwelling element are combined to form the current focus position prompt information of the target interface element.

[0136] The operation result description information and the current focus position prompt information are concatenated into a broadcast text sequence of the target interface element, and the speech synthesis parameters of the broadcast text sequence are configured to obtain the speech broadcast signal of the intelligent enrollment platform.

[0137] The generated target operation instructions are encapsulated according to the message format agreed upon by the operation execution interface and sent to the operation execution interface of the intelligent enrollment platform through the instruction transmission link inside the platform. After receiving the instructions, the operation execution interface calls the interactive execution logic corresponding to the target interface element to complete the instruction execution. All the result data generated by the execution is encapsulated in a unified data format and sent back to the service processing node. The service processing node receives the execution result data returned by the operation execution interface.

[0138] The received execution result data is decomposed and its content is extracted. The operation status identifier representing the execution status of the instruction, the result data payload carrying the specific execution content, and the current interface focus position coordinates representing the current interface focus position are separated from the data. The three types of extracted content are integrated into the interface feedback information of the target interface element.

[0139] Read the operation status identifier in the interface feedback information, determine the corresponding execution status category according to the preset status classification rules, and at the same time read the result data payload in the interface feedback information, determine the corresponding feedback content type according to the preset content classification rules.

[0140] Using the determined execution status category and feedback content type as search criteria, a template matching search is performed on the preset broadcast template library. Broadcast content templates that match both the execution status category and feedback content type are selected. Then, the specific data content in the result data payload is filled into the reserved content position of the broadcast content template to generate the operation result description information of the target interface element.

[0141] Retrieve the stored interface-gesture association information, search for the corresponding association record based on the virtual focus identifier of the target interface element, extract the bound focus coordinate value from the association record, and use the focus coordinate value as the focus position coordinate of the target interface element.

[0142] Based on the focus position coordinates of the target interface element, the focus coordinate values ​​of all interactive elements in the interface-gesture association information are traversed. Interactive elements whose spatial distance from the current focus coordinates is less than a preset neighborhood threshold are selected as the focus dwelling elements of the target interface element. The default neighborhood threshold is 200 pixels. The dwelling elements are sorted according to the direction of top, bottom, left, and right. The element name and element function description of each focus dwelling element are extracted and combined to generate the current focus position prompt information of the target interface element.

[0143] The operation result description information and the current focus position prompt information are concatenated according to the preset broadcast order to form a complete broadcast text sequence of the target interface element. Then, the broadcast text sequence is configured with speech synthesis parameters according to the platform's preset speech broadcast standard, and the speech synthesis engine is called to convert the broadcast text sequence into an audio format speech broadcast signal to obtain the speech broadcast signal of the intelligent enrollment platform.

[0144] The beneficial effects are that by submitting the target operation command to the operation execution interface and obtaining the execution result data, the status information and content data of the interface feedback can be accurately extracted. Combined with the matching and retrieval of the broadcast template library, the operation result description information can be generated in a standardized and clear manner. At the same time, relying on the interface-gesture association information, the focus dwelling element can be located and the position prompt information can be generated. Finally, a complete voice broadcast signal is synthesized, which can provide users with synchronous feedback on the operation result and the current focus position in blind operation scenarios, ensuring that users can grasp the interface interaction status without visual inspection, and improving the smoothness and accuracy of gesture blind operation.

[0145] F. Output the voice broadcast signal, collect the user's confirmation gesture data, and verify the validity of the confirmation gesture data to obtain the user's blind operation assistance result.

[0146] In this embodiment of the invention, the step of validating the confirmation gesture data to obtain the user's blind operation assistance result includes:

[0147] The user's confirmation gesture data is obtained, and the confirmation gesture data is analyzed in multiple dimensions to obtain the confirmation trajectory amplitude value, confirmation trajectory direction change sequence, confirmation trajectory outline type, and confirmation gesture duration.

[0148] The starting direction vector and ending direction vector of the confirmation gesture data are determined based on the confirmed trajectory direction change sequence, and the trajectory turning angle of the confirmation gesture data is determined based on the starting direction vector and the ending direction vector.

[0149] The sequence of confirmed trajectory direction changes is compared item by item with the reference direction vector sequence of the target interface element to obtain the direction matching degree value of the confirmed gesture data;

[0150] The amplitude value of the confirmed trajectory is compared and analyzed with the amplitude reference value of the target interface element to obtain the amplitude matching degree value of the confirmed gesture data;

[0151] The matching degree of the confirmed trajectory contour type and the preset confirmed gesture contour type is measured to obtain the contour matching degree value of the confirmed gesture data, and the duration matching degree value between the duration of the confirmed gesture and the preset standard duration range is calculated.

[0152] The comprehensive verification value of the confirmation gesture data is calculated based on the direction matching value, the amplitude matching value, the contour matching value, and the duration matching value.

[0153] Based on the comprehensive verification value, the operation execution result corresponding to the target operation instruction is used as the blind operation assistance result for the user.

[0154] The formula for calculating the comprehensive verification value is as follows:

[0155] ;

[0156] in, This represents the comprehensive verification value. This indicates the direction matching degree value. This indicates the amplitude matching degree value. This represents the contour matching degree value. This indicates the duration matching degree value. This indicates the preset duration adjustment coefficient.

[0157] The platform's voice playback module converts the generated voice broadcast signal into audio content for playback, while the gesture acquisition device continues to acquire user hand movement data in real time.

[0158] Once the gesture acquisition device detects that the user has completed a complete gesture, it extracts the complete trajectory data corresponding to that gesture as the user's confirmation gesture data.

[0159] The confirmation gesture data is analyzed in multiple dimensions, and the straight-line distance between the first and last sampling points of the trajectory is calculated to obtain the amplitude value of the confirmation trajectory.

[0160] The motion direction corresponding to each group of adjacent sampling points in the confirmation gesture data is extracted sequentially and arranged according to the order of sampling time to obtain the sequence of changes in the direction of the confirmation trajectory.

[0161] The overall motion characteristics of the gesture data are identified to determine the corresponding confirmation trajectory contour type.

[0162] The duration of the confirmation gesture is obtained by statistically analyzing the time span from the first sampling point to the last sampling point.

[0163] Extract the motion direction corresponding to the first sampling point in the confirmation trajectory direction change sequence, and use it as the starting direction vector of the confirmation gesture data.

[0164] Extract the motion direction corresponding to the last sampling point in the confirmation trajectory direction change sequence, and use it as the termination direction vector of the confirmation gesture data.

[0165] By comparing the directional deviation between the starting direction vector and the ending direction vector, the trajectory turning angle of the confirmed gesture data is calculated.

[0166] Retrieve the preset reference direction vector sequence corresponding to the target interface element, and compare the confirmed trajectory direction change sequence with the reference direction vector sequence one by one according to the corresponding position.

[0167] The proportion of items with the same direction among all matched items is used to obtain the direction matching degree value of the confirmed gesture data.

[0168] Retrieve the preset amplitude reference value corresponding to the target interface element, and compare the confirmed trajectory amplitude value with the amplitude reference value.

[0169] The amplitude matching value of the confirmed gesture data is calculated based on the degree of numerical deviation between the two.

[0170] The identified confirmation trajectory contour type is matched with the preset confirmation gesture contour type to obtain the contour matching degree value of the confirmation gesture data.

[0171] Retrieve the preset standard duration range and compare the duration of the confirmation gesture with the standard duration range.

[0172] The duration matching value is calculated based on whether the duration of the confirmed gesture falls within the standard duration range and the degree of deviation from the range.

[0173] The molecular value is obtained by multiplying the orientation matching value, amplitude matching value, and contour matching value.

[0174] The arithmetic mean of the direction matching degree value, amplitude matching degree value and contour matching degree value is taken to obtain the shape matching average value.

[0175] Multiply the duration matching value by the preset duration adjustment coefficient to obtain the duration weighted value. Multiply the average form matching value by the difference between one and the duration adjustment coefficient to obtain the form weighted value. Add the duration weighted value and the form weighted value to obtain the denominator value. Divide the numerator value by the denominator value to obtain the comprehensive verification value of the confirmed gesture data.

[0176] Multiply the direction matching degree value, amplitude matching degree value, and contour matching degree value to obtain the numerator value. Add the direction matching degree value, amplitude matching degree value, and contour matching degree value and divide by three to obtain the average of the three matching degrees. Multiply the duration matching degree value by the duration adjustment coefficient to obtain the duration dimension weighted value. Subtract the duration adjustment coefficient from the value to obtain the remaining weight coefficient. Multiply the average of the three matching degrees by the remaining weight coefficient to obtain the morphology dimension weighted value. Add the duration dimension weighted value and the morphology dimension weighted value to obtain the denominator value. Divide the numerator value by the denominator value to obtain the final comprehensive verification value.

[0177] The direction matching degree, amplitude matching degree, contour matching degree, and duration matching degree are all normalized to the range of 0-1. The higher the value, the higher the matching degree of the corresponding dimension. The duration adjustment coefficient ranges from 0.2 to 0.4, with a default value of 0.3, and is used to adjust the weight ratio of the duration dimension in the comprehensive verification. The baseline direction vector sequence, amplitude baseline value, preset confirmation gesture contour type, and standard duration interval are all pre-collected and calibrated through predefined standard confirmation gestures. The standard duration interval is set to 0.5s-2s by default, and the amplitude baseline value is the straight-line distance between the beginning and end of the trajectory of the standard confirmation gesture.

[0178] The comprehensive verification value is used to quantitatively evaluate the degree of matching between the confirmation gesture data and the benchmark confirmation gesture from multiple dimensions. It integrates the matching information of four dimensions: direction, amplitude, outline, and duration to achieve accurate verification of the validity of the confirmation gesture. The duration adjustment coefficient balances the verification weight of the duration dimension with the other three dimensions to ensure the rationality and accuracy of the verification results. The calculation result serves as the basis for determining the validity of the confirmation gesture and is ultimately used to output the user's blind operation assistance results.

[0179] When the direction matching degree value increases, the comprehensive verification value increases synchronously; when the amplitude matching degree value increases, the comprehensive verification value increases synchronously; when the contour matching degree value increases, the comprehensive verification value increases synchronously; when the duration matching degree value increases, the comprehensive verification value increases synchronously; when the duration adjustment coefficient increases, the influence of the duration matching degree value on the comprehensive verification value increases accordingly; when the duration adjustment coefficient decreases, the influence of the average of the three matching degrees on the comprehensive verification value increases accordingly; when the matching degree values ​​of direction, amplitude, and contour all increase synchronously, the increase in the comprehensive verification value is greater than the increase brought about by the increase in the matching degree of a single dimension.

[0180] The calculated comprehensive verification value is compared with the preset verification threshold. When the comprehensive verification value reaches the preset verification threshold, the gesture is deemed valid, and the operation execution result corresponding to the target operation instruction is determined as the user's blind operation assistance result.

[0181] The beneficial effects are that by performing multi-dimensional analysis and comprehensive verification of confirmation gesture data, the validity of confirmation gestures can be accurately determined from multiple dimensions such as direction, amplitude, outline, and duration, avoiding accidental triggering and invalid operations, ensuring the accuracy of user confirmation actions in blind operation scenarios, and ultimately outputting reliable blind operation assistance results, thereby improving the operation security and interaction stability of gesture interaction.

[0182] like Figure 2 The diagram shown is a functional block diagram of a blind operation assistance system for an intelligent enrollment platform based on gesture interaction, provided in an embodiment of the present invention.

[0183] The gesture-based intelligent enrollment platform blind operation assistance system described in this invention can be installed in an electronic device. Depending on the functions implemented, the gesture-based intelligent enrollment platform blind operation assistance system may include an information acquisition module, an interface-gesture association module, a trajectory recognition module, an information matching module, a voice broadcast module, and an operation assistance module. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and is stored in the memory of the electronic device.

[0184] In this embodiment, the functions of each module / unit are as follows:

[0185] The information collection module is used to respond to the user's entry into the intelligent enrollment platform, collect the user's identity information, and retrieve the interface element distribution configuration data corresponding to the user based on the identity information.

[0186] The interface-gesture association module is used to generate virtual focus positioning information corresponding to interactive elements in the intelligent enrollment platform based on the interface element distribution configuration data, and to map the virtual focus positioning information to the gesture recognition coordinate system of the intelligent enrollment platform to obtain the interface-gesture association information of the intelligent enrollment platform.

[0187] The trajectory recognition module is used to collect the user's dynamic gesture trajectory data in real time, perform trajectory shape recognition on the dynamic gesture trajectory data, and obtain the user's gesture intention feature vector.

[0188] The information matching module is used to match and compare the gesture intention feature vector with the virtual focus positioning information based on the interface-gesture association information, and determine the user's target interface element and target operation instruction based on the comparison result.

[0189] The voice broadcast module is used to apply the target operation command, obtain the interface feedback information of the target interface element, and convert the interface feedback information into a voice broadcast signal. The voice broadcast signal includes operation result description information and current focus position prompt information.

[0190] The operation assistance module is used to output the voice broadcast signal, collect the user's confirmation gesture data, and verify the validity of the confirmation gesture data to obtain the user's blind operation assistance result.

[0191] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0192] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0194] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0195] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A blind operation assistance method for an intelligent enrollment platform based on gesture interaction, characterized in that, The method includes: A. In response to the user's entry into the intelligent enrollment platform, collect the user's identity information and retrieve the interface element distribution configuration data corresponding to the user based on the identity information; B. Based on the interface element distribution configuration data, generate virtual focus positioning information corresponding to the interactive elements in the intelligent enrollment platform, and map the virtual focus positioning information to the gesture recognition coordinate system of the intelligent enrollment platform to obtain the interface-gesture association information of the intelligent enrollment platform, including: The interface position parameters and interface size parameters of the interactive elements are extracted from the interface element distribution configuration data. The interface position parameters include the horizontal and vertical coordinate values ​​of the interactive element in the operation interface of the intelligent enrollment platform. The interface size parameters include the width and height values ​​of the interactive element. The interface position parameters and interface size parameters are processed to obtain the virtual focus candidate positions of the interactive elements by performing center point positioning processing. Assign a unique virtual focus identifier to the interactive element, and associate the virtual focus identifier with the virtual focus candidate position to obtain the virtual focus positioning record of the interactive element; According to the interface hierarchy order in the interface element distribution configuration data, the virtual focus positioning records are arranged to obtain the virtual focus positioning information of the intelligent enrollment platform; Using the top left corner of the intelligent enrollment platform as the origin, with the horizontal direction to the right as the x-axis and the vertical direction downward as the y-axis, a gesture recognition coordinate system for the intelligent enrollment platform is constructed. Based on the coordinate origin position and coordinate axis direction definition of the gesture recognition coordinate system, the virtual focus candidate position is transformed from the interface coordinate system to the gesture recognition coordinate system to obtain the focus coordinate value of the interactive element; Assign corresponding gesture recognition region boundary parameters to the focus coordinate values; The focus coordinate value, the gesture recognition area boundary parameter, and the virtual focus identifier are associated and stored to obtain the interface-gesture association information of the intelligent enrollment platform; C. Collect the user's dynamic gesture trajectory data in real time, perform trajectory shape recognition on the dynamic gesture trajectory data, and obtain the user's gesture intention feature vector; D. Based on the interface-gesture association information, the gesture intention feature vector is matched and compared with the virtual focus positioning information, and the user's target interface element and target operation command are determined according to the comparison result; E. Apply the target operation instruction to obtain the interface feedback information of the target interface element, and convert the interface feedback information into a voice broadcast signal, wherein the voice broadcast signal includes operation result description information and current focus position prompt information; F. Output the voice broadcast signal, collect the user's confirmation gesture data, and verify the validity of the confirmation gesture data to obtain the user's blind operation assistance result.

2. The blind operation assistance method for the intelligent enrollment platform based on gesture interaction as described in claim 1, characterized in that, The action triggered by a user entering the intelligent enrollment platform involves collecting the user's identity information and retrieving the interface element distribution configuration data corresponding to the user based on the identity information, including: The system receives an entry request signal triggered when a user enters the intelligent enrollment platform, and extracts the user's identity information from the entry request signal. The identity information includes the user's login account information and device fingerprint information. Based on the login account information, the user configuration database of the intelligent enrollment platform is retrieved to obtain the user's interface configuration record. The interface configuration record includes the user's custom interface layout parameters and historical operation preference tags. Based on the device fingerprint information, the interface configuration records are matched and filtered to obtain the user's interface element distribution configuration data.

3. The blind operation assistance method for the intelligent enrollment platform based on gesture interaction as described in claim 1, characterized in that, The real-time acquisition of the user's dynamic gesture trajectory data, and the performance of trajectory shape recognition on the dynamic gesture trajectory data to obtain the user's gesture intent feature vector, includes: Acquire the user's dynamic gesture trajectory data, which includes a sequence of coordinate values ​​for trajectory sampling points; The coordinate value sequence is cleaned, and the cleaned coordinate value sequence is interpolated and filled to obtain the user's effective trajectory point sequence; Analyze the direction changes of adjacent trajectory sampling points in the effective trajectory point sequence to obtain the user's trajectory direction change sequence; The distance between the first and last sampling points in the effective trajectory point sequence is taken as the trajectory amplitude value of the effective trajectory point sequence; Contour recognition is performed on the effective trajectory point sequence to obtain the trajectory contour type of the effective trajectory point sequence, and the trajectory contour type includes straight line, arc and polyline; The user's gesture intent feature vector is obtained by fusing the trajectory direction change sequence, the trajectory amplitude value, and the trajectory contour type.

4. The blind operation assistance method for the intelligent enrollment platform based on gesture interaction as described in claim 1, characterized in that, The step of matching and comparing the gesture intent feature vector with the virtual focus positioning information based on the interface-gesture association information, and determining the user's target interface element and target operation command based on the comparison result, includes: Extract the focus coordinates and gesture recognition area boundary parameters of the interactive elements from the interface-gesture association information, and extract the trajectory endpoint coordinates from the gesture intent feature vector. Distance analysis is performed on the focal coordinates and the trajectory endpoint coordinates to obtain the spatial distance between the focal coordinates and the trajectory endpoint coordinates; Based on the spatial distance value and the gesture recognition area boundary parameters, the matching degree value of the interactive element is determined, and the interactive element with the highest matching degree value is selected as the user's target interface element. The element type attribute of the target interface element and the trajectory contour type in the gesture intention feature vector are encoded into the user's target operation command.

5. The blind operation assistance method for the intelligent enrollment platform based on gesture interaction as described in claim 1, characterized in that, The application of the target operation instruction obtains the interface feedback information of the target interface element and converts the interface feedback information into a voice broadcast signal. The voice broadcast signal includes operation result description information and current focus position prompt information, including: Submit the target operation instruction to the operation execution interface of the intelligent enrollment platform, and receive the execution result data returned by the operation execution interface; Information is extracted from the execution result data to obtain the interface feedback information of the target interface element. The interface feedback information includes the operation status identifier, the result data payload, and the coordinates of the current interface focus position. The interface feedback information is parsed to obtain the execution status category and feedback content type of the interface feedback information; Based on the execution status category and the feedback content type, a template matching search is performed on the preset broadcast template library to obtain the broadcast content template of the target interface element, and the specific data content in the result data payload is filled into the broadcast content template to obtain the operation result description information of the target interface element; Extract the focus coordinate value corresponding to the target interface element from the interface-gesture association information to obtain the focus position coordinates of the target interface element; Based on the focus position coordinates, the interface-gesture association information is traversed to obtain the focus dwelling element of the target interface element, and the element name and element function description of the focus dwelling element are combined to form the current focus position prompt information of the target interface element. The operation result description information and the current focus position prompt information are concatenated into a broadcast text sequence of the target interface element, and the speech synthesis parameters of the broadcast text sequence are configured to obtain the speech broadcast signal of the intelligent enrollment platform.

6. The blind operation assistance method for the intelligent enrollment platform based on gesture interaction as described in claim 1, characterized in that, The step of validating the confirmation gesture data to obtain the user's blind operation assistance result includes: The user's confirmation gesture data is obtained, and the confirmation gesture data is analyzed in multiple dimensions to obtain the confirmation trajectory amplitude value, confirmation trajectory direction change sequence, confirmation trajectory outline type, and confirmation gesture duration. The starting direction vector and ending direction vector of the confirmation gesture data are determined based on the confirmed trajectory direction change sequence, and the trajectory turning angle of the confirmation gesture data is determined based on the starting direction vector and the ending direction vector. The sequence of confirmed trajectory direction changes is compared item by item with the reference direction vector sequence of the target interface element to obtain the direction matching degree value of the confirmed gesture data; The amplitude value of the confirmed trajectory is compared and analyzed with the amplitude reference value of the target interface element to obtain the amplitude matching degree value of the confirmed gesture data; The matching degree of the confirmed trajectory contour type and the preset confirmed gesture contour type is measured to obtain the contour matching degree value of the confirmed gesture data, and the duration matching degree value between the duration of the confirmed gesture and the preset standard duration range is calculated. The comprehensive verification value of the confirmation gesture data is calculated based on the direction matching value, the amplitude matching value, the contour matching value, and the duration matching value. Based on the comprehensive verification value, the operation execution result corresponding to the target operation instruction is used as the blind operation assistance result for the user.

7. The blind operation assistance method for the intelligent enrollment platform based on gesture interaction as described in claim 6, characterized in that, The formula for calculating the comprehensive verification value is as follows: ； in, This represents the comprehensive verification value. This indicates the direction matching degree value. This indicates the amplitude matching degree value. This represents the contour matching degree value. This indicates the duration matching degree value. This indicates the preset duration adjustment coefficient.

8. A blind operation assistance system for an intelligent enrollment platform based on gesture interaction, characterized in that: The system is used to implement the blind operation assistance method for the intelligent enrollment platform based on gesture interaction as described in claim 1, the system comprising: The information collection module is used to respond to the user's entry into the intelligent enrollment platform, collect the user's identity information, and retrieve the interface element distribution configuration data corresponding to the user based on the identity information. The interface-gesture association module is used to generate virtual focus positioning information corresponding to interactive elements in the intelligent enrollment platform based on the interface element distribution configuration data, and to map the virtual focus positioning information to the gesture recognition coordinate system of the intelligent enrollment platform to obtain the interface-gesture association information of the intelligent enrollment platform. The trajectory recognition module is used to collect the user's dynamic gesture trajectory data in real time, perform trajectory shape recognition on the dynamic gesture trajectory data, and obtain the user's gesture intention feature vector. The information matching module is used to match and compare the gesture intention feature vector with the virtual focus positioning information based on the interface-gesture association information, and determine the user's target interface element and target operation instruction based on the comparison result. The voice broadcast module is used to apply the target operation command, obtain the interface feedback information of the target interface element, and convert the interface feedback information into a voice broadcast signal. The voice broadcast signal includes operation result description information and current focus position prompt information. The operation assistance module is used to output the voice broadcast signal, collect the user's confirmation gesture data, and verify the validity of the confirmation gesture data to obtain the user's blind operation assistance result.

Images