Human factor intelligent user gaze analysis method, device and system and edge computing equipment

Abstract

The invention discloses a human factor intelligent user gaze analysis method, device and system and edge computing equipment, and belongs to the technical field of computer vision. The method comprises the following steps: collecting visual data of a visual field range of a wearing user through a camera of the head-mounted equipment, and collecting eye movement data of the user in the visual field range through an eye tracker of the head-mounted equipment; identifying a target object in the visual data; determining target eye movement data associated with the fixation of the target object in the eye movement data; and sending the target eye movement data to the target screen so as to correspondingly present the target eye movement data on a target object displayed on the target screen. According to the embodiment of the invention, the user visual behavior analysis efficiency is improved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a human factors intelligent user gaze analysis method, device and system, and edge computing equipment. Background Art

[0002] Eye tracking technology is widely used in psychology, medicine, advertising analysis, autonomous driving and other fields. It tracks the user's eye movements to understand the user's attention distribution, decision-making behavior, reaction speed, etc.

[0003] Traditional eye tracking systems use cameras installed on displays or devices to capture the movement of users' eyes and analyze their visual focus. However, this technology focuses more on post-data analysis, and the mapping of eye movement information on the target screen is done manually by annotators, which is less efficient in analyzing user visual behavior. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a human factors intelligent user gaze analysis method, device and system, and edge computing device to improve the efficiency of user visual behavior analysis.

[0005] In a first aspect, the present application provides a human factors intelligent user gaze analysis method, comprising: Collecting visual data within a field of view of a user wearing the head-mounted device through a camera of the head-mounted device, and collecting eye movement data of the user within the field of view through an eye tracker of the head-mounted device; identifying a target object in the visual data; Determining target eye movement data associated with the gaze of the target object in the eye movement data; The target eye movement data is sent to a target screen so that the target eye movement data is correspondingly presented on a target object displayed on the target screen.

[0006] According to the human factors intelligent user gaze analysis method of the present application, the visual data of the user's field of view is collected through the camera of the head-mounted device, and the eye movement data of the user in the field of view is collected through the eye tracker of the head-mounted device; the target object in the visual data is identified; the target eye movement data associated with the gaze of the target object in the eye movement data is determined; the target eye movement data is sent to the target screen, so that the target eye movement data is presented correspondingly on the target object displayed on the target screen. The embodiment of the present application realizes the effective identification of the target object that the user is looking at by collecting visual data and eye movement data, and displays the user's target eye movement data of the target object on the target screen. The present application does not need to manually mark the user's target gaze data about the target object, and can display the user's target eye movement data of the target object on the target screen in real time, so as to perform real-time analysis of the user's visual behavior on the target screen, thereby enhancing the user's interactive experience and improving the efficiency of the user's visual behavior analysis.

[0007] According to one embodiment of the present application, the method further includes: Determine the two-dimensional logo to be used according to the scenario; Deploy the two-dimensional markers at key points of the target object, and record the corresponding relationship between the target object, the key points and the two-dimensional markers; The identifying the target object in the visual data includes: The position of the target object in the visual data is located according to the position of the two-dimensional mark in the visual data and the corresponding relationship.

[0008] In this embodiment, by determining a suitable two-dimensional marker according to the requirements of the scene, the selection of the two-dimensional marker can be changed according to the scene, so that the two-dimensional marker has sufficient recognition and information capacity in the visual data in different scenes, and the two-dimensional marker is deployed at the key points of the target object, and the corresponding relationship is recorded, which provides necessary reference information for subsequent positioning, so that the position of the target object can be accurately calculated based on the position of the two-dimensional marker in the visual data, thereby improving the accuracy of positioning.

[0009] According to an embodiment of the present application, determining the two-dimensional mark to be used according to the scenario includes: Determine the size of the blank area in the two-dimensional mark according to the scene; wherein the blank area is the area between the edge of the two-dimensional mark and the background area of ​​the two-dimensional mark; The two-dimensional mark to be used is determined according to the size of the blank area.

[0010] In this embodiment, by determining the appropriate size of the blank area according to the scene, the risk of misidentification can be reduced because the blank area provides an additional visual buffer to help distinguish the two-dimensional logo from the surrounding environment, thereby improving the accuracy of two-dimensional logo recognition.

[0011] According to an embodiment of the present application, determining the two-dimensional mark to be used according to the scenario includes: Determining an area ratio of a background area to the two-dimensional logo in the two-dimensional logo according to the scene; A two-dimensional marker to be used is determined according to the area ratio.

[0012] In this embodiment, by precisely controlling the area ratio of the background area to the logo itself in the two-dimensional logo, the two-dimensional logo will not be too prominent in the image to form too strong a contrast with the background, nor will it be too hidden to be difficult to detect, which helps to improve the efficiency of the algorithm in processing images, because it can lock the target logo faster and reduce the processing of irrelevant background information. In addition, the appropriate area ratio can also improve the stability of the logo under different lighting and viewing angle conditions, so that reliable target positioning can be achieved under changing environmental conditions.

[0013] According to an embodiment of the present application, the target object includes a plurality of key points, and the deploying the two-dimensional identification on the key points of the target object includes: Selecting a plurality of non-collinear key points from the target object; deploying the two-dimensional markers on the plurality of non-collinear key points respectively, and recording the corresponding relationship between the target object, the key points and the two-dimensional markers; Alternatively, multiple key points are selected from the target object to form a polygon; the two-dimensional markers are respectively deployed on the multiple key points forming the polygon, and the corresponding relationship between the target object, the key points and the two-dimensional marker is recorded.

[0014] In this embodiment, by selecting multiple non-collinear key points on the target object and deploying two-dimensional markers respectively, the geometric distribution advantage of non-collinear key points is utilized. Even if some markers cannot be identified due to occlusion or damage, the target object can still be located through other visible markers, thereby reducing the impact of single marker failure on the overall positioning accuracy. By utilizing the geometric characteristics of polygons to provide a stable reference frame for positioning, the vertices of the polygons (i.e., key points) are used as the deployment positions of the two-dimensional markers. Even if some markers cannot be identified temporarily due to environmental factors (such as occlusion, light changes), they can still be effectively located based on other visible markers, thereby improving the accuracy of positioning.

[0015] According to an embodiment of the present application, locating the target object according to the position of the two-dimensional mark in the visual data and the corresponding relationship includes: Extracting features from the visual data; detecting the two-dimensional marker included in the visual data and the position of the two-dimensional marker in the visual data according to the extracted features; locating the target object in the visual data according to the corresponding relationship and the detection results of the two-dimensional marker and the position of the two-dimensional marker in the visual data; Alternatively, in response to detecting that the number of two-dimensional markers in the visual data is less than the number of deployed two-dimensional markers, fitting the positions of the undetected two-dimensional markers according to the geometric positional relationship of the key points; The target object is located according to the position of the two-dimensional marker detected in the visual data, the position of the undetected two-dimensional marker obtained by fitting, and the corresponding relationship.

[0016] In this embodiment, by extracting features from visual data, two-dimensional markers in the visual data can be identified and their positions in the visual data can be detected. Combined with the corresponding relationship, the target object can be accurately located. When the number of two-dimensional markers detected in the visual data is less than the number of deployed two-dimensional markers, the position information of the two-dimensional markers that have been detected and the geometric position relationship of the key points can be used to predict the possible positions of the undetected two-dimensional markers by fitting, so that even if the two-dimensional markers are missing, the two-dimensional marker layout can be reconstructed, and the target object can be effectively located, thereby improving the accuracy of positioning.

[0017] According to one embodiment of the present application, the target screen includes a field of view partition and a target object partition; The visual field partitions display the positions of the visual data and the eye movement data in the visual data; The target object partition displays the location of the target object and the target eye movement data on the target object.

[0018] In this embodiment, by dividing the target screen into a field of view partition and a target object partition, the field of view partition setting enables the relative position of the user's eye movement data in his entire field of view to be intuitively displayed, which is helpful for analyzing the user's visual focus distribution and visual path, while the target object partition further focuses on the specific target that the user is looking at. By displaying the specific position of the target eye movement data on the target object, the user's attention allocation and attention details to the target object can be more accurately understood, which not only enhances the visual analysis of the user's gaze behavior, but also can provide guidance for designing user interfaces, optimizing advertising layouts, and improving user experience.

[0019] According to an embodiment of the present application, sending the target eye movement data to the target screen so as to correspondingly present the target eye movement data on the target object displayed on the target screen includes: Acquire position information of the target eye movement data on the target object; Identify the visual data to obtain position information of each marker pre-marked on the target object; Determine the conversion relationship according to the position information of each marker; The position information of the target eye movement data on the target object is substituted into the conversion relationship to obtain the target coordinates of the target eye movement data on the target object in the target screen.

[0020] In this embodiment, by identifying visual data, the position information of each identified marker helps to locate the specific position of the target object and the identification area containing the target eye movement data. Acquiring the position information of the target eye movement data helps to accurately map the target eye movement data onto the target screen. The mapping relationship between the target object and the target screen can be established through the position information of the marker, which helps to accurately align the target object and the target screen during the mapping process, and different marker layouts can adapt to different screen sizes and shapes. After determining the conversion relationship, the position information of the target eye movement data can be mapped to the target screen in real time, which supports interaction with the target object and can see the corresponding response on the target screen, thereby improving the real-time performance of target point mapping.

[0021] According to an embodiment of the present application, determining the conversion relationship according to the position information of each marker includes: Determine, according to the position information of each marker, a first parameter set for calculating the abscissa of the target coordinates, a second parameter set for calculating the ordinate of the target coordinates, and a third parameter set for calculating a normalization factor of the homogeneous coordinates; Determine an expression of a first intermediate variable according to the third parameter set; Obtaining a first expression for calculating the horizontal coordinate of the target coordinate according to the expression of the first intermediate variable and the first parameter set; According to the expression of the first intermediate variable and the second parameter set, a second expression for calculating the ordinate of the target coordinate is obtained, and the first expression and the second expression constitute the conversion relationship.

[0022] In this embodiment, the parameter sets of the normalization factors of the horizontal coordinate, the vertical coordinate and the homogeneous coordinate are respectively determined, which can more accurately describe the mapping relationship from the target object to the target screen. The introduction of intermediate variables can simplify the final conversion relationship expression, making the calculation process more efficient. Through the combination of the horizontal coordinate and the vertical coordinate expressions, a complete conversion relationship is formed, which ensures the position consistency of the target eye movement data between the target object and the target screen. After the conversion relationship is determined, it can be reused multiple times, which improves the real-time and reusability of the system.

[0023] According to an embodiment of the present application, the number of the markers pre-marked on the target object is four, and the corresponding position information is respectively a first coordinate, a second coordinate, a third coordinate and a fourth coordinate; The determining, according to the position information of each marker, a first parameter set for calculating the horizontal coordinate of the target coordinate, a second parameter set for calculating the vertical coordinate of the target coordinate, and a third parameter set for calculating the homogeneous coordinate normalization factor comprises: calculating a second intermediate variable based on the second coordinate, the third coordinate, and the fourth coordinate; calculating a third intermediate variable based on the second intermediate variable, the first coordinate, the second coordinate, and the fourth coordinate; calculating a fourth intermediate variable based on the second intermediate variable, the first coordinate, the third coordinate, and the fourth coordinate; Determine the first parameter set and the second parameter set based on the first coordinate, the second coordinate, the third coordinate, the third intermediate variable, and the fourth intermediate variable; The third parameter set is calculated based on the third intermediate variable and the fourth intermediate variable.

[0024] In this embodiment, a stable framework can be formed by four markers to reduce mapping errors caused by differences in target object shape, size, resolution, etc., can adapt to different types of configurations, has good versatility, and the calculation process is based on clear mathematical expressions and intermediate variables and is easy to implement.

[0025] According to an embodiment of the present application, determining the first parameter set and the second parameter set based on the first coordinate, the second coordinate, the third coordinate, the third intermediate variable, and the fourth intermediate variable includes: The product of the difference between the abscissa of the third coordinate and the first coordinate and the third intermediate variable is used as the first parameter, the product of the difference between the abscissa of the second coordinate and the first coordinate and the fourth intermediate variable is used as the second parameter, and the abscissa of the first coordinate is used as the third parameter, and the first parameter, the second parameter and the third parameter constitute the first parameter set; The product of the difference between the ordinates of the third coordinate and the first coordinate and the third intermediate variable is taken as the fourth parameter, the product of the difference between the ordinates of the second coordinate and the first coordinate and the fourth intermediate variable is taken as the fifth parameter, and the ordinate of the first coordinate is taken as the sixth parameter. The fourth parameter, the fifth parameter and the sixth parameter constitute the second parameter set.

[0026] In this embodiment, accurate mapping results can be provided by calculating parameters based on the position information of four markers through specific mathematical operations, such as multiplication and difference. Difference operations help capture the relative position relationship between different markers. Using multiple markers as reference points and calculating parameter sets based on these reference points can improve the stability of the system. Even if there is an error in the position detection of one of the markers, it can be corrected through the information of other markers, thereby reducing the impact of the error on the mapping results.

[0027] According to an embodiment of the present application, the step of identifying the visual data to obtain the position information of each marker pre-marked on the target object includes: Identifying, from the visual data, a marking area of ​​each marking object pre-marked on the target object; Determine the center point coordinates of each marking area in a coordinate system established based on the visual data to obtain the position information of each marking object; Determining the coordinates of the center point of each marked area includes: For each marked area in the visual data, the coordinates of each point in the outline of the marked area are determined, and the average coordinates of the points in the outline of the marked area are calculated to obtain the coordinates of the center point of the marked area.

[0028] In this embodiment, by using a coordinate system established based on an image, the marker can be accurately located, the accuracy of the position information is ensured, the marking area of ​​the marker is identified and the center point coordinates are determined, and the actual position of the marker in the image can be accurately reflected. By calculating the average coordinates of all points on the contour as the center point coordinates, the overall shape and size of the marking area are taken into account, and the actual position of the marker can be more accurately reflected. Compared with the method of only considering some points or specific points on the contour, the average coordinate method can reduce errors and improve the accuracy of the position information.

[0029] According to an embodiment of the present application, sending the target eye movement data to the target screen so as to correspondingly present the target eye movement data on the target object displayed on the target screen includes: Acquire a first image block including the target eye movement data in a first video frame of the visual data; In response to the first image block satisfying a preset condition, calculating a coordinate mapping relationship between the first video frame and a second video frame in the target screen according to the first image block; The target eye movement data is mapped to the second video frame according to the coordinate mapping relationship.

[0030] In this embodiment, by acquiring image blocks containing target eye movement data, the coordinate mapping relationship between the first video frame and the second image frame is calculated when the image blocks meet preset conditions, so that the calculation process pays more attention to the area in the image related to the target eye movement data rather than the entire first video frame, thereby reducing the impact of environmental changes on the calculation process, improving the accuracy of the coordinate mapping relationship calculation, and thus improving the accuracy of eye movement tracking.

[0031] According to an embodiment of the present application, the first video frame of the visual data is obtained, which includes a first image block of the target eye movement data, including: Determining a position of the target eye movement data in the first video frame; An area within a first preset range centered at the position of the target eye movement data in the first video frame is determined as the first image block.

[0032] In this embodiment, by accurately identifying the position of the target eye movement data and determining the first image block based on the position of the target eye movement data, irrelevant areas in the processed image can be reduced so that the calculation process can focus on the area near the target eye movement data, thereby reducing invalid calculations in irrelevant areas and improving calculation efficiency and accuracy.

[0033] According to one embodiment of the present application, The preset conditions include: the number of matching points in the image block is greater than or equal to a preset number, and / or the multiple matching points in the image block are not collinear; and / or, The method further includes: in response to the first image block not satisfying a preset condition, acquiring a second image block in the first video frame including the target eye movement data; and / or, The method further includes: in response to the second image block satisfying a preset condition, calculating a coordinate mapping relationship between the first video frame and a second video frame in the target screen according to the second image block.

[0034] In this embodiment, a preset condition is set to require that the number of matching points is greater than or equal to a preset number, and these matching points are not collinear, so that there are enough matching points to calculate the coordinate mapping relationship. The condition that multiple matching points are not collinear enables the matching points to provide sufficient geometric information to solve the coordinate mapping relationship between the first video frame and the second video frame, thereby further improving the accuracy of the target eye movement data mapping.

[0035] In this embodiment, by obtaining the second image block including the target eye movement data in the first video frame when the first image block does not meet the preset conditions, even if a sufficient number of matching points or a suitable distribution of matching points cannot be found within the range of the first image block, the possibility of finding valid matching points can be increased by adjusting the search area, thereby enhancing the flexibility and success rate of the target eye movement data mapping process.

[0036] According to an embodiment of the present application, obtaining a second image block including the target eye movement data in the first video frame includes: Acquire the second image block according to the position of the first image block in the first video frame; or, Determine the position of the target eye movement data in the first video frame; determine an area within a second preset range centered on the position of the target eye movement data in the first video frame as the second image block; wherein the second preset range is larger than the first preset range.

[0037] In this embodiment, by acquiring the second image block according to the position of the first image block in the first video frame, a new search range can be re-determined with the first image block as a reference, thereby improving the efficiency of the target eye movement data mapping calculation process.

[0038] In this embodiment, by expanding the first image block into the second image block, even if a sufficient number or a suitable distribution of matching points cannot be found within the initial search range, the possibility of finding valid matching points can be increased by expanding the search area, thereby enhancing the flexibility and success rate of the target eye movement data mapping process.

[0039] According to one embodiment of the present application, before obtaining a first image block including the target eye movement data in a first video frame of the visual data, the method includes: Feature point matching is performed on the first video frame and the second video frame in the target screen, and in response to failure of matching between the first video frame and the second video frame, a video frame is reselected from the visual data to replace the first video frame and the second video frame for feature point matching.

[0040] In this embodiment, when feature point matching fails to successfully establish an accurate correspondence between the first video frame and the second video frame, a new video frame can be flexibly selected from the continuous video stream as the first video frame, thereby reducing the risk of interruption of the entire eye tracking process due to a single matching failure and improving the continuity of the system.

[0041] According to an embodiment of the present application, reselecting a video frame from the visual data to replace the first video frame and the second video frame for feature point matching includes: A video frame adjacent to the first video frame in the visual data or another video frame in the visual data is determined as a reselected video frame.

[0042] In this embodiment, when the feature point matching between the first video frame and the second video frame fails, a video frame adjacent to the current first video frame is selected from the target video stream as a replacement to re-perform feature point matching, so that the reselected video frame maintains temporal continuity, or other video frames are selected as replacements, thereby improving the stability and accuracy of the eye tracking process.

[0043] According to an embodiment of the present application, matching the first video frame with the second video frame in the target screen includes: inputting the first video frame and the second video frame into a preset neural network model so that the neural network model extracts feature points of the first video frame and the second video frame, and matching based on the extracted feature points; or, The feature point matching of the first video frame and the second video frame in the target screen includes: preprocessing the first video frame and the second video frame; the preprocessing includes removing feature points within a preset range of the boundary of the first video frame, and removing feature points within a preset range of the boundary of the second video frame; matching feature points of the preprocessed first video frame and the second video frame; and / or inputting the preprocessed first video frame and the second video frame into a preset neural network model, so that the neural network model can extract feature points of the preprocessed first video frame and the second video frame, and match them according to the extracted feature points.

[0044] In this embodiment, by using a preset neural network model to match the feature points of the first video frame and the second video frame, the powerful feature extraction capability of deep learning can be used to improve the accuracy and efficiency of matching.

[0045] In this embodiment, by removing feature points within a preset range of the image boundary, the negative impact of noise or incompleteness that may exist in the image boundary area on the matching accuracy can be reduced, so that more focus can be placed on more stable and information-rich areas in the image, thereby improving the matching accuracy.

[0046] According to one embodiment of the present application, the method further includes: Storing feature points of the second video frame extracted by the neural network model; Reselecting a video frame from the visual data to replace the first video frame and the second video frame for feature point matching includes: The replaced first video frame is input into a preset neural network model so that the neural network model extracts feature points of the first video frame and matches the feature points of the first video frame with the stored feature points of the second video frame.

[0047] In this embodiment, the features of the second video frame are extracted and stored through a neural network model. When other video frame images are matched with the second video frame, the feature points of the second video frame can be called for feature matching. There is no need to repeatedly extract the features of the second video frame, thereby improving processing efficiency.

[0048] According to an embodiment of the present application, calculating the coordinate mapping relationship between the first video frame and the second video frame in the target screen according to the first image blocks includes: Obtaining first position coordinates of a matching point in the first image block in the first video frame and second position coordinates of the matching point in the second video frame; A homography matrix corresponding to the matching point is calculated according to the first position coordinates and the second position coordinates; wherein the homography matrix represents the coordinate mapping relationship.

[0049] In this embodiment, by calculating the corresponding position coordinates of the matching points in the first video frame and the second video frame, a mathematical model can be established to describe the spatial relationship between these matching points, and the homography matrix is ​​determined based on this to represent the coordinate mapping relationship. Representing the coordinate mapping relationship through the homography matrix can adapt to complex image transformation situations, thereby improving the accuracy and robustness of the target eye movement data mapping.

[0050] According to one embodiment of the present application, the method further includes: Obtaining target eye movement data of multiple users regarding the same target object; Analyze multi-user gaze behaviors based on their corresponding target eye movement data.

[0051] In this embodiment, by analyzing the gaze behaviors of multiple users according to the target eye movement data corresponding to the multiple users, the ability to analyze group user behaviors is improved.

[0052] According to an embodiment of the present application, analyzing the gaze behaviors of multiple users according to target eye movement data corresponding to multiple users includes: The target eye movement data corresponding to multiple users are sent to the target screen, so as to be superimposed and presented on the target object displayed on the target screen.

[0053] In this embodiment, by superimposing the target eye movement data corresponding to multiple users on the target screen, researchers and designers can intuitively observe and compare the common gaze points and gaze patterns of different users when they look at the same target object, thereby identifying the common interest areas or hotspots of the user group.

[0054] According to an embodiment of the present application, analyzing the gaze behaviors of multiple users according to target eye movement data corresponding to multiple users includes: Analyzing the eye movement trajectories and / or eye movement point heat maps of the multiple users according to the target eye movement data corresponding to the multiple users, to obtain the primary viewing positions of the multiple users and the habitual operation processes of the multiple users when performing preset operations; The scene is optimized based on the primary viewing positions of multiple users and the customary operating processes of multiple users when performing preset operations.

[0055] In this embodiment, by analyzing the eye movement trajectories and eye movement point heat maps of multiple users, the users' primary viewing positions and habitual operating processes can be revealed, thereby achieving an in-depth understanding of user behavior patterns. This embodiment provides a quantitative means to evaluate and optimize the user's interactive experience with a product or service. By identifying common gaze points and gaze patterns, designers can optimize the interface layout so that key information and functions conform to the user's natural visual flow and operating habits, thereby improving user satisfaction and operating efficiency.

[0056] In a second aspect, the present application provides a human factors intelligent user gaze analysis device, comprising: A collection module, used to collect visual data within a field of view of a user wearing the head-mounted device via a camera of the head-mounted device, and to collect eye movement data of the user within the field of view via an eye tracker of the head-mounted device; A recognition module, used to recognize a target object in the visual data; A determination module, configured to determine target eye movement data associated with the gaze of the target object in the eye movement data; The display module is used to send the target eye movement data to the target screen so as to present the target eye movement data correspondingly on the target object displayed on the target screen.

[0057] According to the human factors intelligent user gaze analysis device of the present application, the visual data of the field of vision of the user wearing the head-mounted device is collected through the camera of the head-mounted device, and the eye movement data of the user in the field of vision is collected through the eye tracker of the head-mounted device; the target object in the visual data is identified; the target eye movement data associated with the gaze of the target object in the eye movement data is determined; the target eye movement data is sent to the target screen, so that the target eye movement data is presented correspondingly on the target object displayed on the target screen. The embodiment of the present application realizes the effective identification of the target object that the user is looking at by collecting visual data and eye movement data, and displays the target eye movement data of the user on the target object on the target screen. The present application does not need to manually mark the user's target gaze data on the target object, and can display the user's target eye movement data on the target object in real time on the target screen, so as to perform real-time analysis of the user's visual behavior on the target screen, thereby enhancing the user's interactive experience and improving the efficiency of the user's visual behavior analysis.

[0058] In a third aspect, the present application provides an edge computing device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the human factors intelligent user gaze analysis method as described in the first aspect above is implemented.

[0059] According to one embodiment of the present application, the edge computing device includes a head-mounted edge computing device; the head-mounted edge computing device includes a camera and an eye tracker.

[0060] In a fourth aspect, the present application provides a human factors intelligent user gaze analysis system, comprising: a target screen and an edge computing device as described in the third aspect above.

[0061] In a fifth aspect, the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the human factors intelligent user gaze analysis method as described in the first aspect above is implemented.

[0062] In a sixth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the human factors intelligent user gaze analysis method as described in the first aspect above.

[0063] In a seventh aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the human factors intelligent user gaze analysis method as described in the first aspect above.

[0064] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects: According to the human factors intelligent user gaze analysis method of the present application, the visual data of the user's field of view is collected through the camera of the head-mounted device, and the eye movement data of the user in the field of view is collected through the eye tracker of the head-mounted device; the target object in the visual data is identified; the target eye movement data associated with the gaze of the target object in the eye movement data is determined; the target eye movement data is sent to the target screen, so that the target eye movement data is presented correspondingly on the target object displayed on the target screen. The embodiment of the present application realizes the effective identification of the target object that the user is looking at by collecting visual data and eye movement data, and displays the user's target eye movement data of the target object on the target screen. The present application does not need to manually mark the user's target gaze data about the target object, and can display the user's target eye movement data of the target object on the target screen in real time, so as to perform real-time analysis of the user's visual behavior on the target screen, thereby enhancing the user's interactive experience and improving the efficiency of the user's visual behavior analysis.

[0065] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 It is a flowchart of a human factors intelligent user gaze analysis method provided in an embodiment of the present application; Figure 2 is a schematic diagram of a two-dimensional mark provided in an embodiment of the present application; Figure 3 is a schematic diagram of the field of view partition and target object partition provided in an embodiment of the present application; Figure 4 is a schematic diagram of a target eye movement data mapping process provided by an embodiment of the present application; Figure 5 is a structural schematic diagram of a human factors intelligent user gaze analysis device provided in an embodiment of the present application; Figure 6 It is a structural diagram of the edge computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0069] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0070] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0071] In the following, in conjunction with the accompanying drawings, the human factors intelligent user gaze analysis method, device and system, and edge computing device provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0072] Among them, the human factors intelligent user gaze analysis method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0073] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or a tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0074] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0075] The human-factor intelligent user gaze analysis method provided in the embodiment of the present application may be executed by an edge computing device or a functional module or functional entity in the edge computing device that can implement the human-factor intelligent user gaze analysis method. The edge computing device mentioned in the embodiment of the present application may be a portable device (such as a mobile phone, a tablet computer), a wearable device (such as a smart watch, a smart bracelet), an eye tracker, an AR (Augmented Reality) / VR (Virtual Reality) / XR (Extended Reality) / MR (MixedReality) device, an edge server, an edge gateway, a vehicle-mounted device, a cloud computing device, etc. The human-factor intelligent user gaze analysis method provided in the embodiment of the present application is described below using the edge computing device as an example of the execution subject.

[0076] like Figure 1 As shown, the human factor intelligent user gaze analysis method includes: step 110, step 120, step 130 and step 140.

[0077] Step 110: collect visual data within the field of view of the user wearing the head-mounted device through a camera of the head-mounted device, and collect eye movement data of the user within the field of view through an eye tracker of the head-mounted device.

[0078] In the embodiment of the present application, the camera of the head mounted device collects visual data within the field of view of the user wearing the head mounted device. The visual data is equivalent to what the user wearing the head mounted device sees, that is, what is within the field of view of the user. The visual data can be an image or a video. The eye movement data can be data related to the user's eye gaze, for example, the eye movement data can be data such as the eye gaze point, gaze direction, and line of sight trajectory.

[0079] In the embodiment of the present application, the eye movement data may be data related to the user's eye movement captured by an eye tracker of a head-mounted device, and the visual data may be data obtained by capturing objects within the user's field of view through a camera of the head-mounted device. For example, the eye tracker may be a wearable eye tracker, a screen-type eye tracker, a head-mounted eye tracker, etc. The eye tracker may capture the real-time state of the eyes, such as pupils, corneal reflections, etc., to obtain eye movement data, and the camera may capture the real-time state of objects within the field of view to obtain field of view data.

[0080] In one scenario, visual data and eye movement data can be collected through a head-mounted device. Specifically, the head-mounted device can be equipped with a camera and an eye tracker. The camera is responsible for collecting images or video data within the user's field of view, and the eye tracker synchronously collects eye movement data such as the user's gaze point, eye movement trajectory, and gaze duration within the field of view.

[0081] Step 120: Identify the target object in the visual data.

[0082] In the embodiments of the present application, the human factors intelligent user gaze analysis method provided in the embodiments of the present application can be applied to a variety of scenarios, such as user gaze analysis in scenarios such as augmented reality, industrial automation, robot navigation, driving simulators, advertising and marketing, education and entertainment, and user experience. Of course, it can also be applied to other scenarios, which are not limited by the embodiments of the present application. Depending on the scene, the target object can also be different objects. For example, in the driving simulator scene, the target object can be a screen; in the multi-inflection point robot arm recognition scene, the target object can be a robot arm; in the robot navigation scene, the target object can be a turning point, intersection, or specific task area on the navigation path; in the advertising and marketing scene, the target object can be a certain product or a certain product advertisement, etc. The embodiments of the present application do not limit the target object.

[0083] The target object is the object of gaze analysis. For example, in a simulated driving scenario, the target object is the central control screen or different areas of the central control screen. Users can complete some operations by gazing at different areas of the screen, such as turning on the air conditioner, playing music, adjusting the driving mode, etc.

[0084] In the embodiments of the present application, conventional image processing techniques can be used to identify the target object based on the visual data. For example, the visual data is prepared through preprocessing steps such as graying and filtering and denoising, and then the image features of the visual data are extracted using edge detection, corner detection and other algorithms, and the target object is separated from the background through image segmentation techniques such as threshold method or region growing method; finally, feature matching and classical classifiers such as support vector machine (SVM) or Adaboost are used to identify and classify the target object, thereby identifying the target object in the visual data and determining the position of the target object in the visual field; a deep learning algorithm can also be used to pre-train a recognition model, and then the recognition model is deployed in practical applications, and the visual data is input into the recognition model to obtain the target object in the visual data output by the model, and the position of the target object in the visual field. Of course, the target object in the visual data can also be identified by any other method, and the embodiments of the present application are not limited to this.

[0085] Step 130: Determine target eye movement data associated with the gaze of the target object in the eye movement data.

[0086] In the embodiment of the present application, the eye tracker can capture the position and movement of the user's eyes, use the reflection generated by light irradiating into the eyes, extract the reflection characteristics of the pupil and cornea, and calculate the direction of sight by analyzing the pupil position and corneal reflection point. Using geometric principles, combined with the eye structure and light path, the specific position of the eye movement data in the visual data can be inferred.

[0087] After identifying the target object in the visual data, the position of the target object in the visual data can be determined. Based on the position of the eye movement data in the visual data and the position of the target object in the visual data, it can be determined whether the eye movement data is on the target object. If the eye movement data is on the target object, the eye movement data can be determined as target eye movement data associated with the gaze of the target object.

[0088] Step 140: Send the target eye movement data to the target screen, so that the target eye movement data is correspondingly presented on the target object displayed on the target screen.

[0089] In the embodiments of the present application, the target screen may be any device with a display function, for example, the target screen may be a computer screen, a mobile phone screen, a central control screen, the target screen may also be a screen on a head-mounted device, or the target screen may also be other screens, and the symmetry of the embodiments of the present application is not limited thereto.

[0090] In an embodiment of the present application, the target screen may display a target object. It should be noted that the target object displayed on the target screen may be a target object photographed from a different visual perspective than the visual data or a target object photographed from the same visual perspective, and the target object displayed on the target screen is not displayed after receiving the target eye movement data, but has been displayed in advance. One of the purposes of an embodiment of the present application is to map the eye movement data of the user's visual data for the target object to the same target object displayed on the target screen, thereby analyzing the user's behavior of looking at the target object.

[0091] In the embodiment of the present application, the target eye movement data can be sent to the target screen, and the target screen displays the target eye movement data on the target object displayed on the target screen, thereby mapping the user's visual data for the target object's eye movement data to the same target object displayed on the target screen, so as to analyze the user's gaze behavior. For example, the position of the target eye movement data on the target object can be indicated by a mark (such as a dot, a cross, etc.).

[0092] Of course, in order to further clearly display the target eye movement data and the position of the target object to the user or observer, information such as gaze time, gaze frequency, and gaze trajectory can also be displayed on the target screen.

[0093] In some embodiments, the target object displayed on the target screen may be a target object identified from the visual data, and the content displayed on the target screen may be the same as the visual data or partially the same as the visual data. For example, the content displayed on the target screen may be a close-up of the target object, that is, the content displayed on the target screen is mainly focused on the target object. The content displayed on the target screen may differ from the visual data in that the viewing angle of the target object is different, or other content including the target object may be displayed.

[0094] In different application scenarios, the target object may be different. For example, in a simulated driving scenario, the target object may be a driving screen; in a multi-inflection point robot arm recognition scenario, the target object may be a robot arm; in a robot navigation scenario, the target object may be a turning point, intersection, or specific mission area on the navigation path, etc. In other application scenarios, the target object may also be other items, advertisements, etc., and the embodiments of the present application do not limit the target object.

[0095] In the simulated driving scenario, for single-user analysis, the central control screen can be used as the target screen, and the user's target eye movement data for the target object can be mapped to the central control screen for display and analysis, or the user's target eye movement data for the target object can be mapped to other videos that include the target object for display and analysis. For multi-user analysis, the visual data and eye movement data collected from different users are different. The target eye movement data of different users for the target object can be displayed on the target screen to analyze the gaze behavior of multiple users for the target object.

[0096] According to the human factors intelligent user gaze analysis method of the present application, the visual data of the user's field of view is collected through the camera of the head-mounted device, and the eye movement data of the user in the field of view is collected through the eye tracker of the head-mounted device; the target object in the visual data is identified; the target eye movement data associated with the gaze of the target object in the eye movement data is determined; the target eye movement data is sent to the target screen, so that the target eye movement data is presented correspondingly on the target object displayed on the target screen. The embodiment of the present application realizes the effective identification of the target object that the user is looking at by collecting visual data and eye movement data, and displays the user's target eye movement data of the target object on the target screen. The present application does not need to manually mark the user's target gaze data about the target object, and can display the user's target eye movement data of the target object on the target screen in real time, so as to perform real-time analysis of the user's visual behavior on the target screen, thereby enhancing the user's interactive experience and improving the efficiency of the user's visual behavior analysis.

[0097] In some embodiments, the method further comprises: Determine the two-dimensional logo to be used according to the scenario; Deploy two-dimensional markers at key points of the target object, and record the corresponding relationship between the target object, key points and two-dimensional markers; Identify objects in visual data, including: The position of the target object in the visual data is located according to the position and correspondence of the two-dimensional markers in the visual data.

[0098] In this embodiment, the two-dimensional mark is a pattern created in a two-dimensional space using a specific encoding method, and the pattern can be used to store and transmit information. The two-dimensional mark can usually be composed of black and white squares or dot matrices, and can be recognized and decoded by scanning devices such as smart phones or scanners to obtain information stored therein. The design of the two-dimensional mark allows more data to be stored in a limited space. Compared with traditional one-dimensional barcodes, the two-dimensional mark provides a higher data density and information capacity. In this embodiment, the two-dimensional mark can be a two-dimensional code, a data matrix, an Aztec Code, or a two-dimensional mark designed according to other encoding methods, and the embodiments of the present application are not limited to this.

[0099] In this embodiment, the two-dimensional logo to be used can be determined according to the scene. For example, different scenes usually correspond to different environments, and the two-dimensional logo to be used can be determined according to the special effects of different scenes. Some scenes may correspond to outdoor environments with strong or complex lighting, so a two-dimensional logo with high contrast and good reflective properties can be selected; some scenes may correspond to environments with cluttered backgrounds, so a two-dimensional logo with a unique pattern design and easy to distinguish can be selected; some scenes may correspond to dynamic environments, so a logo that can be quickly identified and is insensitive to blur and motion can be selected, etc.

[0100] In some embodiments, the type of two-dimensional mark that matches the scene can be selected. In this embodiment, the type of two-dimensional mark can represent different types of two-dimensional marks, for example, the type of two-dimensional mark can be a two-dimensional code, a data matrix, an Aztec Code, or a two-dimensional mark designed according to other encoding methods.

[0101] The type of the two-dimensional marker can also identify two-dimensional markers of different forms in the same type of two-dimensional marker. For example, if the two-dimensional marker is an ArUco (Augmented Reality University of Cordoba) code, the type of the two-dimensional marker can be the marking mode of the ArUco code. For example, the ArUco library provides a variety of different marking modes, which are mainly distinguished by the number of squares and the number of sub-blocks in each square. Among them, the ArUco code is a binary square reference marker. The ArUco code consists of a wide black background and an internal white binary matrix. Its size and the size of the internal matrix determine the identifier of the marker, that is, each ArUco code has a unique ID. In addition to ID identification, the ArUco code can also be used to accurately estimate the position and direction of the camera relative to the marker.

[0102] The key points of an object can be those locations on the object that have unique geometric features, are easy to identify, and are relatively consistent under different viewing angles. Key points can be corners, edge intersections, center points, or other significant geometric locations of an object. For example, in a multi-screen recognition scenario of a driving simulator, key points can be the corners of the screen, and in a multi-inflection point robotic arm recognition scenario, key points can be joints on the robotic arm. The selection of key points is crucial for subsequent recognition and positioning because key points provide a stable position reference frame.

[0103] In this embodiment, after the key point is selected, a two-dimensional marker may be deployed on the key point, for example, by pasting the two-dimensional marker onto the key point.

[0104] In this embodiment, a database or data structure may be pre-created to store the correspondence between the target object, key points, and two-dimensional identifiers. For each key point, its specific position on the target object is recorded, such as coordinate information, and then each two-dimensional identifier is associated with the specific position deployed on the key point, and the correspondence is recorded. For example, the correspondence may be: target object A-key point a1-two-dimensional identifier a11, target object A-key point a2-two-dimensional identifier a21, target object B-key point b1-two-dimensional identifier b11, etc.

[0105] In this embodiment, a computer vision algorithm, such as edge detection, feature matching or pattern recognition, can be used to identify the specific pattern and structure of the two-dimensional mark, thereby detecting the two-dimensional mark in the visual data. After the two-dimensional mark is detected, the two-dimensional mark can be decoded to obtain information in the two-dimensional mark, which can be coded data, such as numbers, text, etc. It should be noted that the corresponding decoding method can be selected according to the type of the two-dimensional mark for decoding.

[0106] Take the use of edge detection algorithms to detect the position of two-dimensional logos in visual data as an example. In this example, the visual data is image data. After acquiring the image data, the gradient of each pixel in the image can be calculated by the edge detection algorithm. For example, a filter (such as the Sobel operator or the Canny operator) can be applied to the image to calculate the gradient of each pixel in the image. Then, the pixels are sorted according to the gradient size. The algorithm can cluster pixels with similar gradient directions and sizes, and calculate the best straight line through the least squares method. These straight lines represent the edges in the image. Different contours can be identified through the edges in the image. If a contour is similar to the contour of the two-dimensional logo, the location of the contour can be determined as the location of the two-dimensional logo in the image.

[0107] After obtaining the decoded information of the two-dimensional mark, the corresponding relationship of the two-dimensional mark can be searched in the database according to the decoded information, so as to determine the associated target object and key point according to the corresponding relationship. The position of the two-dimensional mark in the visual data is the position of the key point corresponding to the two-dimensional mark, and further, the target object can be located according to the position of the key point.

[0108] In this embodiment, by determining a suitable two-dimensional marker according to the requirements of the scene, the selection of the two-dimensional marker can be changed according to the scene, so that the two-dimensional marker has sufficient recognition and information capacity in the visual data in different scenes, and the two-dimensional marker is deployed at the key points of the target object, and the corresponding relationship is recorded, which provides necessary reference information for subsequent positioning, so that the position of the target object can be accurately calculated based on the position of the two-dimensional marker in the visual data, thereby improving the accuracy of positioning.

[0109] In some embodiments, determining the two-dimensional mark to be used according to the scenario includes: Determine the size of the blank area in the two-dimensional logo according to the scene; wherein the blank area is the area between the edge of the two-dimensional logo and the background area of ​​the two-dimensional logo; The two-dimensional logo to be used is determined according to the size of the blank area.

[0110] In this embodiment, the two-dimensional mark may include a background area and a blank area. Figure 2 As shown, the background area is the area where the code is located. The background area may include a black background and a white binary matrix inside the black background. Different white binary matrices indicate different information contained in the two-dimensional logo. The blank area is the area between the edge of the two-dimensional logo and the edge of the background area. The blank area is a blank area without code. The blank area is crucial for the recognition of the two-dimensional logo because it provides the necessary visual buffer to help the scanning device determine the boundary of the two-dimensional logo.

[0111] In this embodiment, the size of the blank area can be determined according to the characteristics of the scene. The larger the blank area, the larger the distance between the edge of the two-dimensional logo and the edge of the background area. For example, in a strong light or reflective environment, a larger blank area can help the scanning device better identify the boundary of the two-dimensional logo; in a close-range recognition scene, a smaller blank area helps to improve the visibility and recognition rate of the logo. By testing the simulated driving scene, it was found that when the distance between the outer edge of the blank area and the edge of the background area was 0.1cm, it was found that the robustness was poor in different driving scenes. Later, the distance between the outer edge of the blank area and the edge of the background area was increased to 0.5cm, and the recognition rate was greatly improved. Finally, the test found that when the distance between the outer edge of the blank area and the edge of the background area was 0.5-1cm, the recognition accuracy was the highest.

[0112] In this embodiment, the two-dimensional markers of different scenes and different blank areas can be tested and calibrated in advance to determine the blank areas with good positioning effects corresponding to each scene, and establish associations. After determining the scene, the size of the blank area corresponding to the scene can be determined according to the associations, thereby determining the two-dimensional marker to be used.

[0113] In this embodiment, by determining the appropriate size of the blank area according to the scene, the risk of misidentification can be reduced because the blank area provides an additional visual buffer to help distinguish the two-dimensional logo from the surrounding environment, thereby improving the accuracy of two-dimensional logo recognition.

[0114] In some embodiments, determining the two-dimensional mark to be used according to the scenario includes: Determine the area ratio of the background area and the two-dimensional logo in the two-dimensional logo according to the scene; The two-dimensional mark to be used is determined according to the area ratio.

[0115] In this embodiment, the characteristics of the usage scenario can be analyzed in detail, and factors that may be considered include lighting conditions, distance to the target object, background complexity, dynamic changes, etc. According to the characteristics of the scene, the area ratio of the background area to the logo itself in the two-dimensional logo is determined. The area ratio is crucial to the visibility and ease of identification of the two-dimensional logo. For example, in a complex background, the area of ​​the background area can be increased so that it is easier to distinguish the two-dimensional logo from the background in the image. After determining the area ratio, the two-dimensional logo can be designed so that the two-dimensional logo can be accurately identified in different environments. Specifically, during the design process, the size, color, pattern, etc. of the logo can be adjusted to ensure its performance in a specific scenario.

[0116] In this embodiment, two-dimensional markers of different scenes and different area ratios can be tested and calibrated in advance to determine the area ratios with good positioning effects corresponding to each scene, and establish associations. After determining the scene, the area ratio corresponding to the scene can be determined based on the associations, thereby determining the two-dimensional marker to be used. By testing the simulated driving scene, it was found that the area ratio of the background area to the two-dimensional marker in the two-dimensional marker was 0.51-0.73, and the recognition accuracy was the highest.

[0117] In this embodiment, by precisely controlling the area ratio of the background area to the logo itself in the two-dimensional logo, the two-dimensional logo will not be too prominent in the image to form too strong a contrast with the background, nor will it be too hidden to be difficult to detect, which helps to improve the efficiency of the algorithm in processing images, because it can lock the target logo faster and reduce the processing of irrelevant background information. In addition, the appropriate area ratio can also improve the stability of the logo under different lighting and viewing angle conditions, so that reliable target positioning can be achieved under changing environmental conditions.

[0118] In some embodiments, the target object includes a plurality of key points, and two-dimensional markers are deployed on the key points of the target object, including: Select multiple non-collinear key points from the target object; deploy two-dimensional markers on the multiple non-collinear key points respectively, and record the corresponding relationship between the target object, the key points and the two-dimensional markers; Alternatively, multiple key points are selected from the target object to form a polygon; two-dimensional markers are respectively deployed on the multiple key points forming the polygon, and the corresponding relationship between the target object, the key points and the two-dimensional markers is recorded.

[0119] In this embodiment, the target object includes a plurality of key points, and the positions of these key points on the target object are arranged in accordance with a preset position relationship. For example, the preset position relationship may be symmetrical, grid-like, or any other geometric layout that facilitates positioning.

[0120] In this embodiment, the selected multiple key points are not completely collinear. For example, if 3 key points are selected, the 3 key points are not on a straight line; if 4 key points are selected, the 4 key points are not on a straight line.

[0121] In this embodiment, the polygon can be determined according to the number of key points. For example, if the number of key points is 3, the polygon can be a triangle; if the number of key points is 4, the polygon can be a quadrilateral. Depending on the number of key points, the polygon can also be any other polygon, such as a pentagon, a hexagon, etc., which is not limited in the embodiment of the present application.

[0122] In this embodiment, by selecting multiple non-collinear key points on the target object and deploying two-dimensional markers respectively, the geometric distribution advantage of non-collinear key points is utilized. Even if some markers cannot be identified due to occlusion or damage, the target object can still be located through other visible markers, thereby reducing the impact of single marker failure on the overall positioning accuracy. By utilizing the geometric characteristics of polygons to provide a stable reference frame for positioning, the vertices of the polygons (i.e., key points) are used as the deployment positions of the two-dimensional markers. Even if some markers cannot be identified temporarily due to environmental factors (such as occlusion, light changes), they can still be effectively located based on other visible markers, thereby improving the accuracy of positioning.

[0123] In some embodiments, locating the target object according to the position and correspondence of the two-dimensional marker in the visual data includes: Extract features from the visual data; detect the two-dimensional marker included in the visual data and the position of the two-dimensional marker in the visual data according to the extracted features; locate the target object in the visual data according to the corresponding relationship and the detection results of the two-dimensional marker and the position of the two-dimensional marker in the visual data; Alternatively, in response to detecting that the number of two-dimensional markers in the visual data is less than the number of deployed two-dimensional markers, the positions of the undetected two-dimensional markers are fitted based on the geometric position relationship of the key points; and the target object is located based on the positions of the two-dimensional markers detected in the visual data, the fitted positions of the undetected two-dimensional markers and the corresponding relationship.

[0124] In this embodiment, a computer vision algorithm may be used to identify features in an image. For example, a scale-invariant feature transformation algorithm may be used, which extracts features by finding local feature points in an image and calculating descriptors. These feature points are usually corner points, edge points, or other significant image features. The algorithm describes the image blocks around these feature points and generates a feature vector, which can be used to match the same feature points in different images.

[0125] After extracting the features, algorithms can be used to find specific patterns and structures, such as feature matching or pattern recognition of feature points, to identify the outline of the 2D logo. For example, an edge detection algorithm can be used to identify the edges in the image, detect the outline of the 2D logo, and thus determine the position of the 2D logo in the visual data.

[0126] After detecting the position of the two-dimensional marker, the two-dimensional marker can be decoded to obtain the encoded data in the two-dimensional marker, which may include information such as numbers and texts, and then the target object in the visual data can be located in combination with the corresponding relationship.

[0127] In this embodiment, if the number of two-dimensional markers detected in the visual data is less than the number of deployed two-dimensional markers, it may be due to the fact that some two-dimensional markers are blocked or the shooting conditions are not ideal. In this case, the positions of the two-dimensional markers detected in the visual data and the geometric position relationship of the key points can be combined to fit the positions of the undetected two-dimensional markers in the visual data. It should be noted that since the two-dimensional markers are deployed at the key points, the geometric position relationship of the key points is the geometric position relationship of the deployed two-dimensional markers.

[0128] Specifically, the positions of undetected two-dimensional markers can be fitted in any manner. For example, since the arrangement of two-dimensional markers on the target object conforms to the geometric position relationship of key points, when the positions of some two-dimensional markers are known, the possible positions of undetected two-dimensional markers can be fitted by linear fitting, polynomial fitting, geometric fitting, etc.; the geometric position relationship of key points can also be learned by machine learning algorithms, and the positions of undetected two-dimensional markers can be predicted based on the positions of detected two-dimensional markers.

[0129] For example, if two-dimensional markers are deployed on three key points of the target object, a total of three two-dimensional markers are deployed, and the geometric position relationship of the key points is a triangle, when any two two-dimensional markers are detected, the positions of the undetected two-dimensional markers can be fitted according to the geometric relationship of the triangle; if two-dimensional markers are deployed on four key points of the target object, a total of four two-dimensional markers are deployed, and the geometric position relationship of the key points is a quadrilateral, when any three two-dimensional markers are detected, the positions of the undetected two-dimensional markers can be fitted according to the geometric relationship of the quadrilateral. Similarly, when the number of two-dimensional markers is other numbers and the geometric position relationship of the key points is other relationships, the positions of the undetected two-dimensional markers can be fitted according to the positions of the detected two-dimensional markers in the visual data and the geometric position relationship of the key points.

[0130] After fitting the position of the undetected two-dimensional marker, the position of the target object in the visual data can be determined by combining the position of the detected two-dimensional marker in the visual data and the correspondence between the target object, key points and the two-dimensional marker.

[0131] In this embodiment, by extracting features from visual data, two-dimensional markers in the visual data can be identified and their positions in the visual data can be detected. Combined with the corresponding relationship, the target object can be accurately located. When the number of two-dimensional markers detected in the visual data is less than the number of deployed two-dimensional markers, the position information of the two-dimensional markers that have been detected and the geometric position relationship of the key points can be used to predict the possible positions of the undetected two-dimensional markers by fitting, so that even if the two-dimensional markers are missing, the two-dimensional marker layout can be reconstructed, and the target object can be effectively located, thereby improving the accuracy of positioning.

[0132] In some embodiments, the target screen includes a field of view partition and a target object partition; Visual field partitioning shows the location of visual data and eye movement data in the visual data; The target partition displays the location of the target and the target eye movement data on the target.

[0133] In this embodiment, if Figure 3 As shown, the target screen may include a field of view partition and a target object partition, in which the user's visual data and the position of the eye movement data in the visual data may be displayed. In the target object partition, the position of the target object and the target eye movement data on the target object may be displayed.

[0134] In this embodiment, by dividing the target screen into a field of view partition and a target object partition, the field of view partition setting enables the relative position of the user's eye movement data in his entire field of view to be intuitively displayed, which is helpful for analyzing the user's visual focus distribution and visual path, while the target object partition further focuses on the specific target that the user is looking at. By displaying the specific position of the target eye movement data on the target object, the user's attention allocation and attention details to the target object can be more accurately understood, which not only enhances the visual analysis of the user's gaze behavior, but also can provide guidance for designing user interfaces, optimizing advertising layouts, and improving user experience.

[0135] The inventor discovered during the implementation of the embodiments of the present application that in a multi-screen system, when a device with an image / video acquisition function (such as a camera or a device with a camera, which may be referred to as a visual signal acquisition device) is performing image / video acquisition, the shooting angle of the visual signal acquisition device relative to the target object may not be aligned, such as the control position of the visual signal acquisition device is not located directly in front of the target object for shooting, or the shooting angle of the visual signal acquisition device moves or changes during the image / video acquisition process, which causes the image / video captured by the visual signal acquisition device to present a distorted image of the target object, which should have been a standard shape. The distortion of the image makes it difficult to accurately obtain the operation information received on the target object, such as the user's sight point, touch point, mouse click position, etc. on the target object. Therefore, the eye movement data related to the operation information in the distorted sub-image of the target object in the visual data collected by the visual signal acquisition device can be mapped to a non-distorted standard / normal shape target screen, and the eye movement data on the target object in the captured image or video can be presented on the target screen through point mapping, thereby accurately obtaining the operation information received on the target object.

[0136] In some embodiments, sending the target eye movement data to the target screen so as to present the target eye movement data on the target object displayed on the target screen includes: Obtaining the position information of the target eye movement data on the target object; Recognize the visual data to obtain the position information of each marker pre-marked on the target object; Determine the conversion relationship according to the position information of each marker; The position information of the target eye movement data on the target object is substituted into the conversion relationship to obtain the target coordinates of the target eye movement data on the target object in the target screen.

[0137] In this embodiment, referring to the description of step 130, the specific position of the eye movement data in the visual data can be calculated. If the eye movement data falls on the target object, the position information of the eye movement data on the target object can be further determined based on the specific position of the eye movement data in the visual data, that is, the position information of the target eye movement data on the target object can be determined.

[0138] In this embodiment, the marker may be the two-dimensional marker mentioned above, and the two-dimensional marker may be deployed at key points of the target object. By identifying the visual data, the position information of each marker on the target object may be obtained.

[0139] For example, the target object is a screen, and the markers are deployed at the four corners of the screen. The four markers can form a valid quadrilateral. The position information of the markers is the coordinates of the four markers. The distance between each coordinate and the origin of the coordinate system can be calculated. The point with the smallest distance to the origin of the coordinate system is taken as the starting point. The vectors from the starting point to the other three points are calculated, and the angle between each vector and the horizontal coordinate of the coordinate system is determined. The three points are expressed in descending order according to the corresponding angles. , and , the starting point is represented as , according to the coordinates of the four points obtained, the transformation relationship can be determined, and the transformation relationship is used to map the target eye movement data on the distorted rectangle corresponding to the target object to the target screen.

[0140] Specifically, the location information of the target eye movement data can be expressed as , the target coordinates can be expressed as The conversion relationship is an expression containing the position information of the target eye movement data. Substituting the position information of the target eye movement data into it, the target coordinates are obtained, thereby realizing the real-time mapping of the target eye movement data.

[0141] In this embodiment, by identifying visual data, the position information of each identified marker helps to locate the specific position of the target object and the identification area containing the target eye movement data. Acquiring the position information of the target eye movement data helps to accurately map the target eye movement data onto the target screen. The mapping relationship between the target object and the target screen can be established through the position information of the marker, which helps to accurately align the target object and the target screen during the mapping process, and different marker layouts can adapt to different screen sizes and shapes. After determining the conversion relationship, the position information of the target eye movement data can be mapped to the target screen in real time, which supports interaction with the target object and can see the corresponding response on the target screen, thereby improving the real-time performance of target point mapping.

[0142] In some embodiments, determining the conversion relationship according to the position information of each marker includes: Determine, according to the position information of each marker, a first parameter set for calculating the abscissa of the target coordinates, a second parameter set for calculating the ordinate of the target coordinates, and a third parameter set for calculating the normalization factor of the homogeneous coordinates; determining an expression for the first intermediate variable according to the third parameter set; Obtaining a first expression for calculating the abscissa of the target coordinates according to the expression of the first intermediate variable and the first parameter set; According to the expression of the first intermediate variable and the second parameter set, a second expression for calculating the ordinate of the target coordinate is obtained, and the first expression and the second expression form a conversion relationship.

[0143] In this embodiment, based on , and Calculate the second intermediate variable, which can be expressed as ,based on , , and The third intermediate variable can be calculated, and the third intermediate variable can be expressed as ,based on , , and The fourth intermediate variable can be calculated and can be expressed as .

[0144] Furthermore, based on the second intermediate variable , the third intermediate variable and the fourth intermediate variable , the first parameter set, the second parameter set and the third parameter set can be calculated.

[0145] In some embodiments, the expression of the first intermediate variable is:

[0146] in, and are the parameters in the third parameter set, is the location information of the target point; The first expression is:

[0147] in, is the horizontal coordinate of the target coordinate, , and are the parameters in the first parameter set, is the first intermediate variable, The position information of the target eye movement data; The second expression is:

[0148] in, is the ordinate of the target coordinate, , and are the parameters in the second parameter set, is the first intermediate variable, It is the location information of the target eye movement data.

[0149] In this embodiment, the parameter sets of the normalization factors of the horizontal coordinate, the vertical coordinate and the homogeneous coordinate are respectively determined, which can more accurately describe the mapping relationship from the target object to the target screen. The introduction of intermediate variables can simplify the final conversion relationship expression, making the calculation process more efficient. Through the combination of the horizontal coordinate and the vertical coordinate expressions, a complete conversion relationship is formed, which ensures the position consistency of the target eye movement data between the target object and the target screen. After the conversion relationship is determined, it can be reused multiple times, which improves the real-time and reusability of the system.

[0150] In some embodiments, the number of markers pre-marked on the target object is four, and the corresponding position information is respectively the first coordinate, the second coordinate, the third coordinate and the fourth coordinate; Determining, according to the position information of each marker, a first parameter set for calculating the abscissa of the target coordinates, a second parameter set for calculating the ordinate of the target coordinates, and a third parameter set for calculating the normalization factor of the homogeneous coordinates, including: Calculate a second intermediate variable based on the second coordinate, the third coordinate, and the fourth coordinate; Calculate a third intermediate variable based on the second intermediate variable, the first coordinate, the second coordinate, and the fourth coordinate; calculating a fourth intermediate variable based on the second intermediate variable, the first coordinate, the third coordinate, and the fourth coordinate; Determine a first parameter set and a second parameter set based on the first coordinate, the second coordinate, the third coordinate, the third intermediate variable, and the fourth intermediate variable; A third parameter set is calculated based on the third intermediate variable and the fourth intermediate variable.

[0151] In this embodiment, the first coordinate is expressed as , the second coordinate is expressed as , the third coordinate is expressed as , the fourth coordinate is expressed as . According to , and Calculate the second intermediate variable. The expression of the second intermediate variable is:

[0152] Furthermore, according to the second intermediate variable Calculate the third intermediate variable and the fourth intermediate variable , the third intermediate variable The expression is:

[0153] The fourth intermediate variable The expression is:

[0154] The third parameter set includes parameters and parameters , , .

[0155] In this embodiment, a stable framework can be formed by four markers to reduce mapping errors caused by differences in target object shape, size, resolution, etc., can adapt to different types of configurations, has good versatility, and the calculation process is based on clear mathematical expressions and intermediate variables and is easy to implement.

[0156] In some embodiments, determining a first parameter set and a second parameter set based on the first coordinate, the second coordinate, the third coordinate, the third intermediate variable, and the fourth intermediate variable includes: The product of the difference between the abscissa of the third coordinate and the first coordinate and the third intermediate variable is used as the first parameter, the product of the difference between the abscissa of the second coordinate and the first coordinate and the fourth intermediate variable is used as the second parameter, and the abscissa of the first coordinate is used as the third parameter. The first parameter, the second parameter and the third parameter constitute a first parameter set; The product of the difference between the ordinates of the third coordinate and the first coordinate and the third intermediate variable is taken as the fourth parameter, the product of the difference between the ordinates of the second coordinate and the first coordinate and the fourth intermediate variable is taken as the fifth parameter, and the ordinate of the first coordinate is taken as the sixth parameter. The fourth parameter, the fifth parameter and the sixth parameter constitute the second parameter set.

[0157] The first parameter set includes parameters A, B and C, the second parameter set includes parameters D, E and F, and the third parameter set includes parameters G and H. , , , , , .in, and F represent the offset of the target point translation, G and H are used to calculate the normalization factor of the homogeneous coordinates, and the normalization factor is used to maintain the correctness of the perspective transformation.

[0158] In this embodiment, accurate mapping results can be provided by calculating parameters based on the position information of four markers through specific mathematical operations, such as multiplication and difference. Difference operations help capture the relative position relationship between different markers. Using multiple markers as reference points and calculating parameter sets based on these reference points can improve the stability of the system. Even if there is an error in the position detection of one of the markers, it can be corrected through the information of other markers, thereby reducing the impact of the error on the mapping results.

[0159] In some embodiments, the visual data is identified to obtain the position information of each marker pre-marked on the target object, including: Identifying a marking area of ​​each marking object pre-marked on the target object from the visual data; Determine the center point coordinates of each marking area in the coordinate system established based on the visual data to obtain the position information of each marking object; Determine the coordinates of the center point of each marked area, including: For each marked area in the visual data, the coordinates of each point in the outline of the marked area are determined, and the average coordinates of each point in the outline of the marked area are calculated to obtain the coordinates of the center point of the marked area.

[0160] In this embodiment, the lower right corner of the visual data can be set as the coordinate origin, the rightward direction is the positive direction of the x-axis, and the upward direction is the positive direction of the y-axis. Image processing techniques such as edge detection, threshold segmentation, template matching, etc. are used to identify the outline of the marking area of ​​each marker pre-marked on the target object from the visual data.

[0161] In this embodiment, for any identification area in the visual data, the outline of the identification area is represented as the coordinates of a series of points. Each point in the outline is traversed, the number of points in the outline is determined, and the average horizontal coordinate and the average vertical coordinate of each point are calculated. The average horizontal coordinate is the ratio of the sum of the horizontal coordinates of each point to the number of points, and the average vertical coordinate is the ratio of the sum of the vertical coordinates of each point to the number of points. The coordinates of the center point of each identification area can be obtained through the above steps.

[0162] In this embodiment, by using a coordinate system established based on an image, the marker can be accurately located, ensuring the accuracy of the position information, identifying the marker area and determining the center point coordinates, which can accurately reflect the actual position of the marker in the image. By calculating the average coordinates of all points on the contour as the center point coordinates, the overall shape and size of the marker area are taken into account, and the actual position of the marker can be more accurately reflected. Compared with the method of only considering some points or specific points on the contour, the average coordinate method can reduce errors and improve the accuracy of position information. In some embodiments, sending the target eye movement data to the target screen so as to present the target eye movement data on the target object displayed on the target screen includes: Acquire a first image block including target eye movement data in a first video frame of visual data; In response to the first image block satisfying a preset condition, calculating a coordinate mapping relationship between the first video frame and the second video frame in the target screen according to the first image block; The target eye movement data is mapped to the second video frame according to the coordinate mapping relationship.

[0163] In this embodiment, the first image block is a local area of ​​the first video frame, and the target eye movement data is included in the local area.

[0164] In this embodiment, the user is viewing or interacting with an image, such as content on a computer screen, a book page, or a real-world field of view. Through coordinate mapping, the target eye movement data can be mapped to the video frame of the target screen, thereby achieving in-depth analysis and understanding of the user's visual behavior.

[0165] When the first image block meets preset conditions, for example, the quality of the first image block and the information contained therein meet certain conditions, the coordinate mapping relationship between the first video frame and the second video frame can be calculated by image processing and computer vision algorithms.

[0166] In this embodiment, according to the foregoing content, it is known that the position of the target eye movement data in the visual data can be calculated, so that the position of the target eye movement data in the first video frame of the visual data can be obtained. After obtaining the coordinate mapping relationship, the position of the target eye movement data in the first video frame can be calculated based on the coordinate mapping relationship to obtain the position of the target eye movement data in the second video frame of the target screen.

[0167] In this embodiment, by acquiring image blocks containing target eye movement data, the coordinate mapping relationship between the first video frame and the second image frame is calculated when the image blocks meet preset conditions, so that the calculation process pays more attention to the area in the image related to the target eye movement data rather than the entire first video frame, thereby reducing the impact of environmental changes on the calculation process, improving the accuracy of the coordinate mapping relationship calculation, and thus improving the accuracy of eye movement tracking.

[0168] In some embodiments, obtaining a first video frame of visual data including a first image block of target eye movement data includes: Determine the position of the target eye movement data in the first video frame; An area within a first preset range centered on the position of the target eye movement data in the first video frame is determined as a first image block.

[0169] In this embodiment, after determining the position of the target eye movement data in the first video frame, the first image block can be determined according to the position of the target eye movement data. For example, the first image block is an area within a preset range of the position of the target eye movement data in the first video frame, and the first image block can be a circular area, a square area, etc.

[0170] In some embodiments, an area within a first preset range centered on the position of the target eye movement data in the first video frame may be determined as the first image block. The first preset range may be a circular area, a square area, etc. The first preset range is smaller than the first video frame. For example, if the resolution of the first video frame is W*H, the first preset range may be an area centered on the target eye movement data and having a size of 0.1W and 0.1H.

[0171] In this embodiment, by accurately identifying the position of the target eye movement data and determining the first image block based on the position of the target eye movement data, irrelevant areas in the processed image can be reduced so that the calculation process can focus on the area near the target eye movement data, thereby reducing invalid calculations in irrelevant areas and improving calculation efficiency and accuracy.

[0172] In some embodiments, the preset conditions include: the number of matching points in the image block is greater than or equal to a preset number, and / or the multiple matching points in the image block are not collinear.

[0173] In this embodiment, a preset condition is set to require that the number of matching points is greater than or equal to a preset number, and these matching points are not collinear, so that there are enough matching points to calculate the coordinate mapping relationship. The condition that multiple matching points are not collinear enables the matching points to provide sufficient geometric information to solve the coordinate mapping relationship between the first video frame and the second video frame, thereby further improving the accuracy of the target eye movement data mapping.

[0174] In some embodiments, the method further includes: in response to the first image block not satisfying a preset condition, acquiring a second image block including the target eye movement data in the first video frame.

[0175] The method further includes: in response to the second image block satisfying a preset condition, calculating a coordinate mapping relationship between the first video frame and the second video frame in the target screen according to the second image block.

[0176] In this embodiment, if the first image block does not meet the preset conditions, a second image block can be re-determined, the second image block also includes the target eye movement data, and the second image block and the first image block are different local areas in the first video frame. The second image block and the first image block can be image blocks of the same shape, or image blocks of different shapes; the second image block can be an image block obtained by translating the first image block, or an image block obtained by enlarging the first image block.

[0177] In this embodiment, if the second image block meets a preset condition, the coordinate mapping relationship between the first video frame and the second video frame can be calculated according to the second image block.

[0178] In this embodiment, by obtaining the second image block including the target eye movement data in the first video frame when the first image block does not meet the preset conditions, even if a sufficient number or a suitable distribution of matching points cannot be found within the range of the first image block, the possibility of finding valid matching points can be increased by adjusting the search area, thereby enhancing the flexibility and success rate of the target eye movement data mapping process.

[0179] In some embodiments, obtaining a second image block including target eye movement data in a first video frame includes: The second image block is obtained according to the position of the first image block in the first video frame.

[0180] In this embodiment, the second image block may be an image block obtained by translating the first image block, or may be an image block obtained by enlarging the first image block. The second image block may be determined by the position of the first image block.

[0181] In this embodiment, by acquiring the second image block according to the position of the first image block in the first video frame, a new search range can be re-determined with the first image block as a reference, thereby improving the efficiency of the target eye movement data mapping calculation process.

[0182] In some embodiments, obtaining a second image block including target eye movement data in a first video frame includes: Determine the position of the target eye movement data in the first video frame; determine an area within a second preset range centered on the position of the target eye movement data in the first video frame as a second image block; wherein the second preset range is larger than the first preset range.

[0183] In this embodiment, the first preset range can be expanded to a second preset range to obtain a second image block, so as to expand the search range to find enough matching points, thereby improving the accuracy of calculating the coordinate mapping relationship. For example, the first preset range can be expanded to 1.1 times, 1.2 times, 1.5 times, 2 times, or other multiples of the original to obtain the second preset range.

[0184] In this embodiment, by expanding the first image block into the second image block, even if a sufficient number or a suitable distribution of matching points cannot be found within the initial search range, the possibility of finding valid matching points can be increased by expanding the search area, thereby enhancing the flexibility and success rate of the target eye movement data mapping process.

[0185] In some embodiments, before acquiring a first image block including target eye movement data in a first video frame of visual data, the method includes: Feature point matching is performed on the first video frame and the second video frame in the target screen. In response to failure of matching between the first video frame and the second video frame, a video frame is reselected from the visual data to replace the first video frame and the second video frame for feature point matching.

[0186] In this embodiment, feature points are some unique and identifiable points in the image, such as bright spots, spots, corners or other significant visual features. These feature points are equivalent to visual landmarks in the image and are parts of the image that are easily identified and distinguished. Matching points refer to feature points that are the same or similar between two or more images. In tasks such as image registration, stereo vision, image stitching or object recognition, matching points are used to establish correspondence between different images. It should be noted that feature points and matching points can be a pixel point or an area including multiple pixel points.

[0187] Feature point matching is a process of extracting feature points from two or more images and identifying whether these feature points are matching points. Specifically, feature extraction algorithms, such as scale-invariant feature transformation, accelerated robust features, etc., can be used to extract feature points in the first video frame and the second video frame, respectively, and generate descriptors for the feature points. The descriptors can be quantitative representations of the neighborhood around the feature points, which are used to describe the characteristics of the feature points so that they can be identified and matched in different images. Then, points that match the feature points in the first video frame are searched in the second video frame. For example, the descriptors of the feature points in the first video frame and the second video frame can be compared, and the similarity of the feature points can be evaluated through distance metrics, such as Euclidean distance and Hamming distance, and the feature points whose similarity is greater than a preset threshold are determined as matching points.

[0188] In this embodiment, matching points are obtained by matching feature points, so that a spatial relationship between the first video frame and the second video frame can be established, providing an accurate reference benchmark for subsequent image segmentation, and enabling the target eye movement data to be effectively located and mapped to the corresponding position of the second video frame. Furthermore, the determination of matching points also helps to optimize the allocation of computing resources, because one can focus on these matching points without redundantly processing the entire image, thereby improving the efficiency and accuracy of eye tracking.

[0189] When the first video frame and the second video frame fail to match, it means that the similarity between the feature points in the first video frame and the feature points in the second video frame does not reach a preset threshold, and other video frames in the visual data can be selected to replace the first video frame as new first video frames and second video frames for matching.

[0190] In this embodiment, when feature point matching fails to successfully establish an accurate correspondence between the first video frame and the second video frame, a new video frame can be flexibly selected from the continuous video stream as the first video frame, thereby reducing the risk of interruption of the entire eye tracking process due to a single matching failure and improving the continuity of the system.

[0191] In some embodiments, reselecting a video frame from the visual data to replace the first video frame and the second video frame for feature point matching includes: A video frame adjacent to the first video frame in the visual data or another video frame in the visual data is determined as a reselected video frame.

[0192] In this embodiment, a video frame adjacent to the first video frame in the visual data or other video frames in the visual data may be determined as a reselected video frame to continue matching with the second video frame as a new first video frame.

[0193] In this embodiment, when the feature point matching between the first video frame and the second video frame fails, a video frame adjacent to the current first video frame is selected from the target video stream as a replacement to re-perform feature point matching, so that the reselected video frame maintains temporal continuity, or other video frames are selected as replacements, thereby improving the stability and accuracy of the eye tracking process.

[0194] In some embodiments, feature point matching is performed on a first video frame and a second video frame in a target screen, including: inputting the first video frame and the second video frame into a preset neural network model so that the neural network model extracts feature points of the first video frame and the second video frame, and matching is performed based on the extracted feature points.

[0195] In this embodiment, a neural network model can be pre-trained so that the neural network model can recognize and process feature points in the image. For example, the neural network in the neural model can analyze the input first video frame and the second video frame through its multi-layer structure, extract key visual features, including texture, color, shape, edge, etc., and then calculate the similarity of the feature points to output the final matching points.

[0196] In this embodiment, by using a preset neural network model to match the feature points of the first video frame and the second video frame, the powerful feature extraction capability of deep learning can be used to improve the accuracy and efficiency of matching.

[0197] In some embodiments, feature point matching is performed on a first video frame and a second video frame in a target screen, including: preprocessing the first video frame and the second video frame; the preprocessing includes removing feature points within a preset range of the boundary of the first video frame, and removing feature points within a preset range of the boundary of the second video frame; feature point matching is performed on the preprocessed first video frame and the second video frame; and / or, the preprocessed first video frame and the second video frame are input into a preset neural network model, so that the neural network model extracts feature points of the preprocessed first video frame and the second video frame, and matches them according to the extracted feature points.

[0198] In some embodiments, the preprocessed first video frame and the second video frame may be input into a preset neural network model so that the neural network model extracts feature points of the preprocessed first video frame and the second video frame and performs matching based on the extracted feature points.

[0199] In this embodiment, by removing feature points within a preset range of the image boundary, the negative impact of noise or incompleteness that may exist in the image boundary area on the matching accuracy can be reduced, so that more focus can be placed on more stable and information-rich areas in the image, thereby improving the matching accuracy.

[0200] In some embodiments, the method further comprises: Storing feature points of the second video frame extracted by the neural network model; Reselecting a video frame from the visual data to replace the first video frame and the second video frame for feature point matching, including: The replaced first video frame is input into a preset neural network model so that the neural network model extracts feature points of the first video frame and matches the feature points of the first video frame with the stored feature points of the second video frame.

[0201] In this embodiment, since the first video frame and the second video frame may fail to match, if a new video frame is selected as the first video frame and the second video frame for matching after each matching failure, and the feature points of the second video frame are extracted, computing resources will be wasted. Therefore, the feature points of the second video frame can be stored, so that after the matching fails, the feature points of the new first video frame can be extracted, and the stored feature points of the second video frame can be reused without repeatedly extracting the feature points of the second video frame.

[0202] In this embodiment, the features of the second video frame are extracted and stored through a neural network model. When other video frame images are matched with the second video frame, the feature points of the second video frame can be called for feature matching. There is no need to repeatedly extract the features of the second video frame, thereby improving processing efficiency.

[0203] In some embodiments, calculating the coordinate mapping relationship between the first video frame and the second video frame in the target screen according to the first image block includes: Obtaining first position coordinates of a matching point in a first image block in a first video frame and second position coordinates of a matching point in a second video frame; The homography matrix corresponding to the matching point is calculated according to the first position coordinates and the second position coordinates; wherein the homography matrix represents the coordinate mapping relationship.

[0204] The homography matrix describes the projection relationship between different planes. This projection relationship can be used to map points on one plane to corresponding points on another plane, and is usually used in scenes such as image matching, stereo vision, and image stitching. In this embodiment, the coordinate mapping relationship can be represented by the homography matrix, which is used to transform the points in the first video frame into the coordinate system of the second video frame so that the two images can be aligned. For example, is a point in the first video frame, is a point in the second video frame, then and The coordinate mapping relationship can be expressed as: .in represents the homography matrix.

[0205] In this embodiment, the first position coordinates of the matching point in the first image block in the first video frame and the second position coordinates of the matching point in the second video frame can be obtained. For example, there are four matching points in the first image block, namely matching point 1, matching point 2, matching point 3 and matching point 4. The coordinates of matching point 1 in the first video frame are , the coordinates in the second video frame are , the coordinates of matching point 2 in the first video frame are , the coordinates in the second video frame are , the coordinates of matching point 3 in the first video frame are , the coordinates in the second video frame are , the coordinates of matching point 4 in the first video frame are , the coordinates in the second video frame are , and then use the position coordinates of these matching points in different images to construct a linear equation system to calculate the homography matrix :

[0206]

[0207] in, is the homography matrix Elements of .

[0208] Then, the above equations are solved by the least squares method or other numerical methods, such as direct linear transformation, to obtain the homography matrix Elements It can be seen that if the number of matching points is greater than 4 and these matching points are not collinear, the homography matrix can be obtained Therefore, the preset condition can be determined according to the number of matching points and / or the distribution of matching points. For example, the preset condition can be set to be that the number of matching points is greater than or equal to 4, and these matching points are not collinear. Of course, the above calculation process is based on the example of 4 matching points. When the number of matching points is other values, the above calculation process can be adaptively modified according to the number of matching points.

[0209] like Figure 4 As shown, the five-pointed star in the second video frame represents the position of the target eye movement data. After the coordinate mapping relationship is calculated, the position of the target eye movement data can be calculated with the coordinate mapping relationship to obtain the position of the target eye movement data in the second video frame.

[0210] In this embodiment, by calculating the corresponding position coordinates of the matching points in the first video frame and the second video frame, a mathematical model can be established to describe the spatial relationship between these matching points, and the homography matrix is ​​determined based on this to represent the coordinate mapping relationship. Representing the coordinate mapping relationship through the homography matrix can adapt to complex image transformation situations, thereby improving the accuracy and robustness of the target eye movement data mapping.

[0211] In some embodiments, the method further comprises: Obtaining target eye movement data of multiple users regarding the same target object; Analyze multi-user gaze behaviors based on their corresponding target eye movement data.

[0212] In this embodiment, target eye movement data of multiple users gazing at the same target object can be obtained, and the gaze behavior of multiple users can be analyzed based on these target eye movement data. For example, based on these target eye movement data, the concentrated position of each user's gaze on the target object can be analyzed, and the position where each user gazes the longest on the target object can be analyzed, so as to analyze the position that most users are more interested in.

[0213] In this embodiment, by analyzing the gaze behaviors of multiple users according to the target eye movement data corresponding to the multiple users, the ability to analyze group user behaviors is improved.

[0214] In some embodiments, the target eye movement data of a single user can also be analyzed to analyze the user's gaze behavior. For example, the position on the target object where the user gazes the longest, the position where the user gazes the most frequently, etc. can be analyzed based on the user's target eye movement data, thereby analyzing the position on the target object that is most attractive to the user.

[0215] In some embodiments, analyzing the gaze behaviors of multiple users according to target eye movement data corresponding to multiple users includes: The target eye movement data corresponding to the multiple users are sent to the target screen, so as to be superimposed and presented on the target object displayed on the target screen.

[0216] In this embodiment, the target eye movement data of multiple different users can be superimposed and displayed in the target screen. For example, the visual data of multiple different users and the positions of the eye movement data of different users in the visual data can be displayed in the field of view partition. The target object can be displayed in the target object partition, and the positions of the target eye movement data corresponding to multiple users on the target object can be superimposed and displayed on the target object.

[0217] In this embodiment, by superimposing the target eye movement data corresponding to multiple users on the target screen, researchers and designers can intuitively observe and compare the common gaze points and gaze patterns of different users when they look at the same target object, thereby identifying the common interest areas or hotspots of the user group.

[0218] In some embodiments, analyzing the gaze behaviors of multiple users according to target eye movement data corresponding to multiple users includes: Analyzing the eye movement trajectories and / or eye movement point heat maps of the multiple users according to the target eye movement data corresponding to the multiple users, to obtain the primary viewing positions of the multiple users and the habitual operation processes of the multiple users when performing preset operations; The scene is optimized based on the primary viewing positions of multiple users and the customary operating processes of multiple users when performing preset operations.

[0219] In this embodiment, the user's eye movement trajectory can be analyzed through the target eye movement data to identify the user's viewing mode and order, and find out the primary viewing position of different users. By collecting data such as eye movement points and gaze duration of different users, an eye movement point heat map is generated, and the concentrated area of ​​user gaze is displayed through color changes. For example, warm colors such as red and yellow can be used to represent areas with dense gaze or long duration.

[0220] Through eye tracking and heat maps, the user's habitual operation process when performing preset operations is analyzed, and the design is optimized according to the user's primary viewing position and habitual operation process. For example, in user interface design, important information or controls can be placed in the user's primary viewing position to simplify the user's operation process and improve the user experience.

[0221] In this embodiment, by analyzing the eye movement trajectories and eye movement point heat maps of multiple users, the users' primary viewing positions and habitual operating processes can be revealed, thereby achieving an in-depth understanding of user behavior patterns. This embodiment provides a quantitative means to evaluate and optimize the user's interactive experience with a product or service. By identifying common gaze points and gaze patterns, designers can optimize the interface layout so that key information and functions conform to the user's natural visual flow and operating habits, thereby improving user satisfaction and operating efficiency.

[0222] The human factor intelligent user gaze analysis method provided in the embodiment of the present application can be executed by a human factor intelligent user gaze analysis device. In the embodiment of the present application, the human factor intelligent user gaze analysis device executing the human factor intelligent user gaze analysis method is taken as an example to illustrate the human factor intelligent user gaze analysis device provided in the embodiment of the present application.

[0223] The embodiment of the present application also provides a human factors intelligent user gaze analysis device.

[0224] like Figure 5 As shown, the human factor intelligent user gaze analysis device includes: The acquisition module 510 is used to acquire visual data of the user's field of vision through the camera of the head-mounted device, and to acquire eye movement data of the user in the field of vision through the eye tracker of the head-mounted device; Recognition module 520, for recognizing a target object in visual data; A determination module 530 is used to determine target eye movement data associated with the gaze of the target object in the eye movement data; The display module 540 is used to send the target eye movement data to the target screen, so as to present the target eye movement data correspondingly on the target object displayed on the target screen.

[0225] According to the human factors intelligent user gaze analysis device of the present application, the visual data of the user's field of view is collected through the camera of the head-mounted device, and the eye movement data of the user in the field of view is collected through the eye tracker of the head-mounted device; the target object in the visual data is identified; the target eye movement data associated with the gaze of the target object in the eye movement data is determined; the target eye movement data is sent to the target screen, so that the target eye movement data is presented correspondingly on the target object displayed on the target screen. The embodiment of the present application realizes the effective identification of the target object that the user is looking at by collecting visual data and eye movement data, and displays the user's target eye movement data of the target object on the target screen. The present application does not need to manually mark the user's target gaze data about the target object, and can display the user's target eye movement data of the target object on the target screen in real time, so as to perform real-time analysis of the user's visual behavior on the target screen, thereby enhancing the user's interactive experience and improving the efficiency of the user's visual behavior analysis.

[0226] The human factor intelligent user gaze analysis device in the embodiment of the present application can be an edge computing device, or a component in the edge computing device, such as an integrated circuit or a chip. The edge computing device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the edge computing device can be a mobile phone, a tablet computer, a laptop computer, an eye tracker, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited. In an optional example, an edge deployment solution in which the human factors intelligent user gaze analysis device is deployed as an edge computing device or component on the end side can migrate at least part of the computing tasks to the end side device, give full play to the advantages of edge computing to improve data processing efficiency, reduce latency and network resource occupancy and dependence, and also facilitate data security and privacy protection.

[0227] The human factors intelligent user gaze analysis device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0228] In some embodiments, Figure 6 As shown, an embodiment of the present application also provides an edge computing device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, each process of the above-mentioned human factors intelligent user gaze analysis method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0229] It should be noted that the edge computing devices in the embodiments of the present application include the above-mentioned mobile edge computing devices and non-mobile edge computing devices.

[0230] In some embodiments, the edge computing device may be a head-mounted edge computing device; the head-mounted edge computing device may include a camera and an eye tracker.

[0231] In some embodiments, the embodiments of the present application also provide a human factors intelligent user gaze analysis system, including: a target screen and the above-mentioned edge computing device.

[0232] The embodiment of the present application also provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned human factors intelligent user gaze analysis method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0233] The processor is the processor in the edge computing device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0234] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned human factors intelligent user gaze analysis method when executed by a processor.

[0235] The processor is the processor in the edge computing device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0236] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned human factors intelligent user gaze analysis method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0237] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0238] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0239] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0240] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application, all of which are within the protection of the present application.

[0241] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

Claims

1. A human factors intelligent user gaze analysis method, characterized in that: include: Collecting visual data within a field of view of a user wearing the head-mounted device through a camera of the head-mounted device, and collecting eye movement data of the user within the field of view through an eye tracker of the head-mounted device; identifying a target object in the visual data; Determining target eye movement data associated with the gaze of the target object in the eye movement data; The target eye movement data is sent to a target screen so that the target eye movement data is correspondingly presented on a target object displayed on the target screen.

2. The method according to claim 1, characterized in that The method further comprises: Determine the two-dimensional logo to be used according to the scenario; Deploy the two-dimensional markers at key points of the target object, and record the corresponding relationship between the target object, the key points and the two-dimensional markers; The identifying the target object in the visual data includes: The position of the target object in the visual data is located according to the position of the two-dimensional mark in the visual data and the corresponding relationship.

3. The method according to claim 2, characterized in that The determining of the two-dimensional mark to be used according to the scenario includes: Determine the size of the blank area in the two-dimensional mark according to the scene; wherein the blank area is the area between the edge of the two-dimensional mark and the background area of ​​the two-dimensional mark; The two-dimensional mark to be used is determined according to the size of the blank area.

4. The method according to claim 2, characterized in that: The determining of the two-dimensional mark to be used according to the scenario includes: Determining an area ratio of a background area to the two-dimensional logo in the two-dimensional logo according to the scene; A two-dimensional marker to be used is determined according to the area ratio.

5. The method according to claim 2, characterized in that: The target object includes a plurality of key points, and the step of deploying the two-dimensional identification on the key points of the target object includes: Selecting a plurality of non-collinear key points from the target object; deploying the two-dimensional markers on the plurality of non-collinear key points respectively, and recording the corresponding relationship between the target object, the key points and the two-dimensional markers; Alternatively, multiple key points are selected from the target object to form a polygon; the two-dimensional markers are respectively deployed on the multiple key points forming the polygon, and the corresponding relationship between the target object, the key points and the two-dimensional marker is recorded.

6. The method according to claim 2, characterized in that The locating the target object according to the position of the two-dimensional mark in the visual data and the corresponding relationship includes: Extracting features from the visual data; detecting the two-dimensional marker included in the visual data and the position of the two-dimensional marker in the visual data according to the extracted features; locating the target object in the visual data according to the corresponding relationship and the detection results of the two-dimensional marker and the position of the two-dimensional marker in the visual data; Alternatively, in response to detecting that the number of two-dimensional markers in the visual data is less than the number of deployed two-dimensional markers, fitting the positions of the undetected two-dimensional markers according to the geometric positional relationship of the key points; The target object is located according to the position of the two-dimensional marker detected in the visual data, the position of the undetected two-dimensional marker obtained by fitting, and the corresponding relationship.

7. The method according to claim 1, characterized in that The target screen includes a visual field partition and a target object partition; The visual field partitions display the positions of the visual data and the eye movement data in the visual data; The target object partition displays the location of the target object and the target eye movement data on the target object.

8. The method according to claim 1, characterized in that The sending the target eye movement data to the target screen so as to correspondingly present the target eye movement data on the target object displayed on the target screen includes: Acquire position information of the target eye movement data on the target object; Identify the visual data to obtain position information of each marker pre-marked on the target object; Determine the conversion relationship according to the position information of each marker; The position information of the target eye movement data on the target object is substituted into the conversion relationship to obtain the target coordinates of the target eye movement data on the target object in the target screen.

9. The method according to claim 8, characterized in that The step of determining the conversion relationship according to the position information of each marker includes: Determine, according to the position information of each marker, a first parameter set for calculating the abscissa of the target coordinates, a second parameter set for calculating the ordinate of the target coordinates, and a third parameter set for calculating a normalization factor of the homogeneous coordinates; Determine an expression of a first intermediate variable according to the third parameter set; Obtaining a first expression for calculating the horizontal coordinate of the target coordinate according to the expression of the first intermediate variable and the first parameter set; According to the expression of the first intermediate variable and the second parameter set, a second expression for calculating the ordinate of the target coordinate is obtained, and the first expression and the second expression constitute the conversion relationship.

10. The method according to claim 9, characterized in that The number of the markers pre-marked on the target object is four, and the corresponding position information is a first coordinate, a second coordinate, a third coordinate and a fourth coordinate respectively; The determining, according to the position information of each marker, a first parameter set for calculating the horizontal coordinate of the target coordinate, a second parameter set for calculating the vertical coordinate of the target coordinate, and a third parameter set for calculating the homogeneous coordinate normalization factor comprises: calculating a second intermediate variable based on the second coordinate, the third coordinate, and the fourth coordinate; Calculate a third intermediate variable based on the second intermediate variable, the first coordinate, the second coordinate, and the fourth coordinate; calculating a fourth intermediate variable based on the second intermediate variable, the first coordinate, the third coordinate, and the fourth coordinate; Determine the first parameter set and the second parameter set based on the first coordinate, the second coordinate, the third coordinate, the third intermediate variable, and the fourth intermediate variable; The third parameter set is calculated based on the third intermediate variable and the fourth intermediate variable.

11. The method according to claim 10, characterized in that The determining the first parameter set and the second parameter set based on the first coordinate, the second coordinate, the third coordinate, the third intermediate variable, and the fourth intermediate variable includes: The product of the difference between the abscissa of the third coordinate and the first coordinate and the third intermediate variable is used as the first parameter, the product of the difference between the abscissa of the second coordinate and the first coordinate and the fourth intermediate variable is used as the second parameter, and the abscissa of the first coordinate is used as the third parameter, and the first parameter, the second parameter and the third parameter constitute the first parameter set; The product of the difference between the ordinates of the third coordinate and the first coordinate and the third intermediate variable is taken as the fourth parameter, the product of the difference between the ordinates of the second coordinate and the first coordinate and the fourth intermediate variable is taken as the fifth parameter, and the ordinate of the first coordinate is taken as the sixth parameter. The fourth parameter, the fifth parameter and the sixth parameter constitute the second parameter set.

12. The method according to claim 8, characterized in that The step of identifying the visual data to obtain the position information of each marker pre-marked on the target object includes: Identifying, from the visual data, a marking area of ​​each marking object pre-marked on the target object; Determine the center point coordinates of each marking area in a coordinate system established based on the visual data to obtain the position information of each marking object; Determining the coordinates of the center point of each marked area includes: For each marked area in the visual data, the coordinates of each point in the outline of the marked area are determined, and the average coordinates of the points in the outline of the marked area are calculated to obtain the coordinates of the center point of the marked area.

13. The method according to claim 1, characterized in that The sending the target eye movement data to the target screen so as to correspondingly present the target eye movement data on the target object displayed on the target screen includes: Acquire a first image block including the target eye movement data in a first video frame of the visual data; In response to the first image block satisfying a preset condition, calculating a coordinate mapping relationship between the first video frame and a second video frame in the target screen according to the first image block; The target eye movement data is mapped to the second video frame according to the coordinate mapping relationship.

14. The method according to claim 13, characterized in that The first video frame of the visual data is obtained, which includes a first image block of the target eye movement data, including: Determining a position of the target eye movement data in the first video frame; An area within a first preset range centered at the position of the target eye movement data in the first video frame is determined as the first image block.

15. The method according to claim 13, characterized in that The preset conditions include: the number of matching points in the image block is greater than or equal to a preset number, and / or the multiple matching points in the image block are not collinear; and / or, The method further includes: in response to the first image block not satisfying a preset condition, acquiring a second image block in the first video frame including the target eye movement data; and / or, The method further includes: in response to the second image block satisfying a preset condition, calculating a coordinate mapping relationship between the first video frame and a second video frame in the target screen according to the second image block.

16. The method according to claim 15, characterized in that The obtaining of a second image block including the target eye movement data in the first video frame includes: Acquire the second image block according to the position of the first image block in the first video frame; or, Determine the position of the target eye movement data in the first video frame; determine an area within a second preset range centered on the position of the target eye movement data in the first video frame as the second image block; wherein the second preset range is larger than the first preset range.

17. The method according to claim 13, characterized in that Before acquiring a first image block including the target eye movement data in a first video frame of the visual data, the method includes: Feature point matching is performed on the first video frame and the second video frame in the target screen, and in response to failure of matching between the first video frame and the second video frame, a video frame is reselected from the visual data to replace the first video frame and the second video frame for feature point matching.

18. The method according to claim 17, characterized in that The reselecting a video frame from the visual data to replace the first video frame and the second video frame for feature point matching includes: A video frame adjacent to the first video frame in the visual data or another video frame in the visual data is determined as a reselected video frame.

19. The method according to claim 17, characterized in that The matching of feature points between the first video frame and the second video frame in the target screen includes: inputting the first video frame and the second video frame into a preset neural network model so that the neural network model extracts feature points of the first video frame and the second video frame, and matching is performed based on the extracted feature points; or, The feature point matching of the first video frame and the second video frame in the target screen includes: preprocessing the first video frame and the second video frame; the preprocessing includes removing feature points within a preset range of the boundary of the first video frame, and removing feature points within a preset range of the boundary of the second video frame; matching feature points of the preprocessed first video frame and the second video frame; and / or inputting the preprocessed first video frame and the second video frame into a preset neural network model, so that the neural network model can extract feature points of the preprocessed first video frame and the second video frame, and match them according to the extracted feature points.

20. The method according to claim 19, characterized in that The method further comprises: Storing feature points of the second video frame extracted by the neural network model; Reselecting a video frame from the visual data to replace the first video frame and the second video frame for feature point matching includes: The replaced first video frame is input into a preset neural network model so that the neural network model extracts feature points of the first video frame and matches the feature points of the first video frame with the stored feature points of the second video frame.

21. The method according to claim 13, characterized in that The calculating, according to the first image blocks, a coordinate mapping relationship between the first video frame and the second video frame in the target screen includes: Obtaining first position coordinates of a matching point in the first image block in the first video frame and second position coordinates of the matching point in the second video frame; A homography matrix corresponding to the matching point is calculated according to the first position coordinates and the second position coordinates; wherein the homography matrix represents the coordinate mapping relationship.

22. The method according to claim 1, characterized in that The method further comprises: Obtaining target eye movement data of multiple users regarding the same target object; Analyze multi-user gaze behaviors based on their corresponding target eye movement data.

23. The method according to claim 22, characterized in that The analyzing the gaze behaviors of multiple users according to the target eye movement data corresponding to the multiple users includes: The target eye movement data corresponding to multiple users are sent to the target screen, so as to be superimposed and presented on the target object displayed on the target screen.

24. The method according to claim 22, characterized in that The analyzing the gaze behaviors of multiple users according to the target eye movement data corresponding to the multiple users includes: Analyzing the eye movement trajectories and / or eye movement point heat maps of the multiple users according to the target eye movement data corresponding to the multiple users, to obtain the primary viewing positions of the multiple users and the habitual operation processes of the multiple users when performing preset operations; The scene is optimized based on the primary viewing positions of multiple users and the customary operating processes of multiple users when performing preset operations.

25. A human factors intelligent user gaze analysis device, characterized in that: include: A collection module, used to collect visual data within a field of view of a user wearing the head-mounted device via a camera of the head-mounted device, and to collect eye movement data of the user within the field of view via an eye tracker of the head-mounted device; A recognition module, used to recognize a target object in the visual data; A determination module, configured to determine target eye movement data associated with the gaze of the target object in the eye movement data; The display module is used to send the target eye movement data to the target screen so as to present the target eye movement data correspondingly on the target object displayed on the target screen.

26. An edge computing device, characterized in that: It includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the human factors intelligent user gaze analysis method as described in any one of claims 1 to 24 is implemented.

27. The edge computing device according to claim 26, characterized in that: The edge computing device includes a head-mounted edge computing device; the head-mounted edge computing device includes a camera and an eye tracker.

28. A human factors intelligent user gaze analysis system, characterized in that: include: Target screen, edge computing device as described in claim 26 or 27.

29. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the human factors intelligent user gaze analysis method as described in any one of claims 1 to 24 is implemented.

Images