Human factor intelligent dynamic region of interest determination method and device, edge computing equipment and medium

Abstract

The invention discloses a human factor intelligent dynamic region of interest determination method and device, edge computing equipment and a medium, and belongs to the technical field of computer vision. The method comprises the following steps: collecting eye movement data when a user watches a video through an eye movement tracking device; performing instance segmentation on the video frame of the video to obtain a plurality of segmentation areas of the plurality of objects on the video frame; and matching the plurality of segmented regions according to the position of the eye movement data in the video frame so as to determine the region of interest at the position in the video frame. According to the embodiment of the invention, dynamic determination of the region of interest can be realized, and the determination efficiency of the region of interest is improved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a method, device, edge computing device and medium for determining a human-based intelligent dynamic area of ​​interest. Background Art

[0002] Users usually manually define AOI (Area of ​​Interest) through a graphical user interface. For example, using a rectangle or polygon tool, users can directly draw an area of ​​interest on an image or video frame. However, when processing a large amount of video, the manual definition of AOI does not support real-time video dynamics and is difficult to adapt quickly, so it is inefficient. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a human-based intelligent dynamic interest area determination method, device, edge computing device and medium to achieve dynamic determination of interest areas and improve the efficiency of determining interest areas.

[0004] In a first aspect, the present application provides a method for determining a dynamic area of ​​interest based on human intelligence, comprising:

[0005] Collect eye movement data of users watching videos through eye tracking equipment;

[0006] Performing instance segmentation on the video frame of the video to obtain multiple segmented regions of multiple objects on the video frame;

[0007] The multiple segmented areas are matched according to the positions of the eye movement data in the video frame to determine the interest area at the position in the video frame.

[0008] According to the human-factor intelligent dynamic interest area determination method of the present application, the eye movement data of the user watching the video is collected through an eye tracking device; the video frame of the video is segmented by instance to obtain multiple segmented areas of multiple objects on the video frame; the multiple segmented areas are matched according to the position of the eye movement data in the video frame to determine the interest area at the position in the video frame. The embodiment of the present application collects the eye movement data of the user watching the video, and segments the video frame in combination with the instance segmentation technology. By matching the eye movement data with the segmented areas in the video frame, the user's attention can be associated with a specific object or scene in the video, thereby determining the interest area, reducing the tedious process of manual frame-by-frame analysis, and can realize object recognition and tracking across frames and scenes. When the video is played for a long time, the interest area that the user is concerned about can be continuously obtained, thereby realizing dynamic determination of the interest area and improving the efficiency of determining the interest area.

[0009] According to an embodiment of the present application, performing instance segmentation on the video frame of the video to obtain multiple segmented regions of multiple objects on the video frame includes:

[0010] Identifying pixel boundaries of each object and the category of each object in the video frame;

[0011] The video frame is segmented according to the pixel boundaries of each object to obtain a plurality of segmented regions of a plurality of objects on the video frame.

[0012] In this embodiment, the pixel boundaries of each object in the video frame are identified and the video frame is segmented according to the pixel boundaries of each object, so that the segmentation process has a high correlation with the contour of the object, thereby improving the accuracy of object segmentation.

[0013] According to an embodiment of the present application, segmenting the video frame according to the pixel boundaries of each object to obtain multiple segmented regions of multiple objects on the video frame includes:

[0014] Mask the pixels within the pixel boundary range of each object to obtain the mask area of ​​each object;

[0015] The mask area surrounding each object is enclosed by the object bounding box;

[0016] The mask region and / or the object bounding box is determined as a plurality of segmented regions of a plurality of objects on the video frame.

[0017] In this embodiment, by masking the pixel boundaries of the object, clear object mask regions can be generated, and these mask regions correspond to the actual contours of the object, thereby improving the accuracy of segmentation. Furthermore, by enclosing these mask regions with object bounding boxes, not only can the contours of the object be identified, but also the relative position and spatial range of the object in the video frame can be accurately determined, thereby improving the accuracy of subsequent determination of the region of interest.

[0018] According to an embodiment of the present application, matching the multiple segmented regions according to the position of the eye movement data in the video frame to determine the interest area at the position in the video frame includes:

[0019] Matching the position of the eye movement data in the video frame with the position of the mask area of ​​each object in the video frame;

[0020] In the case where the eye movement data is matched to be within the mask area of ​​the target object, the location of the target object is determined as the area of ​​interest.

[0021] In this embodiment, by matching the position of the eye movement data with the position of the mask area of ​​each object in the video frame, it is possible to locate the object that the user is actually looking at. When the eye movement data falls within the mask area of ​​an object, the position of the object can be accurately determined as the area of ​​interest, thereby providing more accurate data support for video content analysis and user behavior research in areas such as personalized video content recommendation and advertising placement.

[0022] According to an embodiment of the present application, matching the position of the eye movement data in the video frame with the position of the mask area of ​​each object in the video frame includes:

[0023] Preliminarily matching the position of the eye movement data in the video frame with the object bounding box of each object;

[0024] When it is preliminarily matched that the eye movement data is in the object bounding box of the target object, it is determined whether the eye movement data is in the mask area of ​​the target object.

[0025] In this embodiment, by first using the object bounding box for preliminary matching, the range of objects that the user's line of sight may focus on can be quickly screened out. After the preliminary match is successful, it is further checked whether the eye movement data falls within the mask area of ​​the target object, reducing the area that requires further precise matching, thereby improving matching efficiency.

[0026] According to an embodiment of the present application, matching the multiple segmented regions according to the position of the eye movement data in the video frame to determine the interest area at the position in the video frame includes:

[0027] In the case that the eye movement data is matched and is not in the mask area of ​​any object, the position of the target object corresponding to the mask area closest to the eye movement data is determined as the area of ​​interest.

[0028] In this embodiment, the eye movement data is compared with the object mask area in the video frame, and the mask area closest to the eye movement data is selected if there is no direct match. This method does not require the user's line of sight to fall completely within the object mask area, but allows a certain degree of line of sight deviation, thereby increasing the flexibility of determining the area of ​​interest.

[0029] According to an embodiment of the present application, matching the position of the eye movement data in the video frame with the position of the mask area of ​​each object in the video frame includes:

[0030] In the case where the eye movement data is matched to an object bounding box of multiple candidate objects,

[0031] The location of the candidate object with the largest object bounding box area is determined as the area of ​​interest;

[0032] Alternatively, different weights are assigned to different candidate objects according to their categories or positions, and the position of the candidate object with the highest weight is determined as the area of ​​interest.

[0033] In this embodiment, by selecting the candidate object with the largest object bounding box area as the region of interest, it is considered that the user may pay more attention to objects with large areas, so these areas may contain more important information. Secondly, by assigning weights based on categories or positions to different candidate objects and selecting the object with the highest weight as the region of interest, the user's visual interest can be predicted based on the importance of the object or the relative position to the user's line of sight, thereby improving the accuracy of region of interest recognition.

[0034] According to one embodiment of the present application, the method further includes:

[0035] Determining a target object corresponding to the area of ​​interest;

[0036] The position of the interest area in the video frame at a future moment is predicted according to the position of the target object in the video frame at a historical moment and the position of the target object in the current video frame.

[0037] In this embodiment, by determining the target object corresponding to the area of ​​interest and using the position of the object in the video frames at historical moments and the current moment, the position of the object at future moments is predicted, thereby optimizing the tracking of the area of ​​interest of the video stream, facilitating continued tracking when the object is obscured or temporarily leaves the field of view, and improving the accuracy of area of ​​interest tracking.

[0038] According to one embodiment of the present application, the method further includes:

[0039] Extracting the position information and shape information of the target object corresponding to the area of ​​interest;

[0040] The target object is drawn on the target screen according to the position information and the shape information.

[0041] In this embodiment, by extracting the position and shape information of the target objects corresponding to the area of ​​interest, these objects can be accurately reproduced on the target screen, and the specific performance of the content that the user is concerned about on the screen can be more intuitively seen, and both the position and shape of the objects can be accurately reflected.

[0042] According to one embodiment of the present application, the method further includes:

[0043] When the area of ​​interest is updated, removing the target object drawn on the target screen;

[0044] The target object is drawn on the target screen according to the updated position information and shape information of the target object corresponding to the region of interest.

[0045] In this embodiment, by promptly removing the target object drawn on the target screen when the area of ​​interest is updated, and redrawing the object according to the position and shape information of the target object corresponding to the new area of ​​interest, the content displayed on the screen is always the latest information of the object, so that the display of video content is more in line with the user's real-time needs and preferences.

[0046] According to one embodiment of the present application, the video includes at least one of the following:

[0047] Video played on a display device;

[0048] A video of the user's field of view captured by a camera of a head-mounted edge computing device worn by the user;

[0049] Video within the user's field of view displayed by a head-mounted edge computing device worn by the user.

[0050] In a second aspect, the present application provides a human-based intelligent dynamic interest area determination device, comprising:

[0051] The acquisition module collects the eye movement data of users watching videos through eye tracking equipment;

[0052] A segmentation module, used to perform instance segmentation on the video frame of the video to obtain multiple segmented regions of multiple objects on the video frame;

[0053] A matching module is used to match the multiple segmented areas according to the position of the eye movement data in the video frame to determine the interest area at the position in the video frame.

[0054] According to the human-factor intelligent dynamic interest area determination device of the present application, the eye movement data of the user watching the video is collected through an eye tracking device; the video frame of the video is segmented by instance to obtain multiple segmented areas of multiple objects on the video frame; the multiple segmented areas are matched according to the position of the eye movement data in the video frame to determine the interest area at the position in the video frame. The embodiment of the present application collects the eye movement data of the user watching the video, and segments the video frame in combination with the instance segmentation technology. By matching the eye movement data with the segmented areas in the video frame, the user's attention can be associated with a specific object or scene in the video, thereby determining the interest area, reducing the tedious process of manual frame-by-frame analysis, and can realize object recognition and tracking across frames and scenes. When the video is played for a long time, the interest area that the user is concerned about can be continuously obtained, thereby realizing dynamic determination of the interest area and improving the efficiency of determining the interest area.

[0055] In a third aspect, the present application provides an edge computing device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the method for determining a human-based intelligent dynamic area of ​​interest as described in the first aspect above is implemented.

[0056] According to one embodiment of the present application, the edge computing device includes a head-mounted edge computing device; the head-mounted edge computing device includes a camera and an eye tracker.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the method for determining a human-based intelligent dynamic interest area as described in the first aspect above is implemented.

[0058] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the human-based intelligent dynamic interest area determination method as described in the first aspect above.

[0059] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the human intelligent dynamic interest area determination method as described in the first aspect above.

[0060] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0061] According to the human-factor intelligent dynamic interest area determination method of the present application, the eye movement data of the user watching the video is collected through an eye tracking device; the video frame of the video is segmented by instance to obtain multiple segmented areas of multiple objects on the video frame; the multiple segmented areas are matched according to the position of the eye movement data in the video frame to determine the interest area at the position in the video frame. The embodiment of the present application collects the eye movement data of the user watching the video, and segments the video frame in combination with the instance segmentation technology. By matching the eye movement data with the segmented areas in the video frame, the user's attention can be associated with a specific object or scene in the video, thereby determining the interest area, reducing the tedious process of manual frame-by-frame analysis, and can realize object recognition and tracking across frames and scenes. When the video is played for a long time, the interest area that the user is concerned about can be continuously obtained, thereby realizing dynamic determination of the interest area and improving the efficiency of determining the interest area.

[0062] In some embodiments, by identifying the pixel boundaries of each object in the video frame, the video frame is segmented according to the pixel boundaries of each object, so that the segmentation process has a high correlation with the contour of the object, thereby improving the accuracy of object segmentation.

[0063] In some embodiments, by masking the pixel boundaries of the object, a clear object mask region can be generated, and these mask regions correspond to the actual contours of the object, thereby improving the accuracy of segmentation. Furthermore, by enclosing these mask regions with object bounding boxes, not only can the contour of the object be identified, but also the relative position and spatial range of the object in the video frame can be accurately determined, thereby improving the accuracy of subsequent determination of the region of interest.

[0064] In some embodiments, by matching the position of the eye movement data with the position of the mask area of ​​each object in the video frame, it is possible to locate the object that the user is actually looking at. When the eye movement data falls within the mask area of ​​an object, the position of the object can be accurately determined as the area of ​​interest, thereby providing more accurate data support for video content analysis and user behavior research in areas such as personalized video content recommendation and advertising placement.

[0065] In some embodiments, by first using the object bounding box for preliminary matching, the range of objects that the user's line of sight may focus on can be quickly screened out. After the preliminary match is successful, the eye movement data is further checked to see if it falls within the mask area of ​​the target object, thereby reducing the area that requires further precise matching, thereby improving matching efficiency.

[0066] In some embodiments, by comparing the eye movement data with the object mask area in the video frame and selecting the mask area closest to the eye movement data if there is no direct match, this method does not require the user's line of sight to fall completely within the object mask area, but allows a certain degree of line of sight deviation, thereby increasing the flexibility of determining the area of ​​interest.

[0067] In some embodiments, by selecting the candidate object with the largest object bounding box area as the region of interest, it is considered that the user may pay more attention to large-area objects, and therefore these areas may contain more important information. Secondly, by assigning category- or location-based weights to different candidate objects and selecting the object with the highest weight as the region of interest, the user's visual interest can be predicted based on the importance of the object or its relative position to the user's line of sight, thereby improving the accuracy of region of interest recognition.

[0068] In some embodiments, by determining the target object corresponding to the region of interest and using the position of the object in the video frames at historical moments and the current moment to predict the position of the object at future moments, the tracking of the region of interest for the video stream is optimized, which helps to continue tracking when the object is obscured or temporarily leaves the field of view, thereby improving the accuracy of region of interest tracking.

[0069] In some embodiments, by extracting the position and shape information of the target objects corresponding to the area of ​​interest, these objects can be accurately reproduced on the target screen, so that the user can more intuitively see the specific performance of the content they are concerned about on the screen, and both the position and shape of the objects can be accurately reflected.

[0070] In some embodiments, by promptly removing the target object drawn on the target screen when the area of ​​interest is updated, and redrawing the object according to the position and shape information of the target object corresponding to the new area of ​​interest, the content displayed on the screen is always the latest information of the object, so that the display of video content is more in line with the user's real-time needs and preferences.

[0071] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 It is a flowchart of a method for determining a dynamic area of ​​interest based on human intelligence provided in an embodiment of the present application;

[0074] Figure 2 is a schematic diagram of an object provided in an embodiment of the present application being displayed on a target screen;

[0075] Figure 3 is a schematic diagram of updating an object in a target screen provided by an embodiment of the present application;

[0076] Figure 4 is a structural schematic diagram of a human-based intelligent dynamic interest area determination device provided in an embodiment of the present application;

[0077] Figure 5 It is a structural diagram of the edge computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0079] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0080] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0081] In combination with the accompanying drawings, the human-based intelligent dynamic interest area determination method, device, edge computing device and medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0082] Among them, the human-based intelligent dynamic interest area determination method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0083] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0084] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0085] The human-factor intelligent dynamic interest area determination method provided in the embodiment of the present application may be executed by an edge computing device or a functional module or functional entity in an edge computing device that can implement the human-factor intelligent dynamic interest area determination method. The edge computing device mentioned in the embodiment of the present application may be a portable device (such as a mobile phone, a tablet computer), a wearable device (such as a smart watch, a smart bracelet), an eye tracker, an AR (Augmented Reality) / VR (Virtual Reality) / XR (Extended Reality) / MR (Mixed Reality) device, an edge server, an edge gateway, a vehicle-mounted device, a cloud computing device, etc. The human-factor intelligent dynamic interest area determination method provided in the embodiment of the present application is described below using an edge computing device as an example of an execution subject.

[0086] like Figure 1 As shown, the human-based intelligent dynamic interest area determination method includes: step 110, step 120 and step 130.

[0087] Step 110: Collect eye movement data of the user watching the video through an eye tracking device.

[0088] In the embodiment of the present application, the eye tracking device is a device for measuring and recording the position and movement of the eyes. The eye tracking device can track the movement and gaze point of the eyes by emitting near-infrared light to the eyes and detecting its reflection on the pupil and cornea based on near-infrared light technology.

[0089] In some embodiments, the eye tracking device may be a head-mounted edge computing device. The video may be a video of the user's field of view captured by a camera of the head-mounted edge computing device worn by the user, and the video may include things seen by the eyes of the user wearing the head-mounted edge computing device, that is, things within the user's field of view. The video may also be a video within the user's field of view displayed by the head-mounted edge computing device. For example, the head-mounted edge computing device may also include a display screen, and the video within the user's field of view is displayed on the display screen. Of course, the video may also be a video played on other display devices, and the embodiments of the present application do not limit the video.

[0090] Eye movement data can be data related to the user's eye movement captured by an eye tracker of a head-mounted edge computing device, for example, eye movement data can be data such as eye gaze point, gaze direction, sight track, gaze duration, etc. The eye tracker can be a wearable eye tracker, a screen-mounted eye tracker, a head-mounted eye tracker, etc. The eye tracker can capture the real-time state of the eyes, such as pupils, corneal reflections, etc. to obtain eye movement data.

[0091] In one scenario, a head-mounted edge computing device may be equipped with a camera and an eye tracker. The camera is responsible for collecting video within the user's field of view, while the eye tracker synchronously collects eye movement data such as the user's gaze point, eye movement trajectory, and gaze duration within the field of view.

[0092] Step 120: perform instance segmentation on the video frame of the video to obtain multiple segmented regions of multiple objects on the video frame.

[0093] Instance segmentation is a task in the field of computer vision. Through instance segmentation, objects in an image can be identified and the outline of each object can be located, that is, each object can be segmented at the pixel level. Instance segmentation combines the characteristics of target detection and semantic segmentation. It can not only identify the classification of objects at the pixel level, but also distinguish different instances of the same category in the image.

[0094] The objects in the video frames are independent objects that appear in each frame of the video, such as monitors, keyboards, cups, mobile phones, notepads, etc. If the user is viewing a web page, the objects can also be objects in the web page, such as buttons, billboards, etc. Different objects appear in different forms and positions in the video frames. Different objects can be identified through instance segmentation, and the corresponding segmentation area can be determined according to the location of the object.

[0095] An example is used to illustrate the process of instance segmentation of a video frame. In this example, target detection algorithms such as YOLO, Faster R-CNN, etc. can be used to identify the position of each object in the video frame, and the position of each object can be represented by a bounding box. The pixels in the bounding box are feature extracted, and a convolutional neural network is used to distinguish different objects. The objects are segmented according to their positions in the video frame to obtain multiple segmented regions, where each object corresponds to a segmented region and each object is inside the corresponding segmented region.

[0096] Step 130: Match multiple segmented areas according to the positions of the eye movement data in the video frame to determine the interest area at the positions in the video frame.

[0097] In the embodiment of the present application, the position and movement of the user's eyes can be captured by an eye tracker, and the reflection generated by light irradiating into the eyes can be used to extract the reflection characteristics of the pupil and cornea, and the direction of the line of sight can be calculated by analyzing the pupil position and corneal reflection point. Using geometric principles, combined with the eye structure and light path, the specific position of the eye movement data in the video frame can be inferred.

[0098] In an embodiment of the present application, the position of the eye movement data in the video frame can be matched with multiple segmented areas. For example, the position coordinates of the eye movement data in the video frame can be compared with the area boundaries of each segmented area to determine which area the eye movement data falls in or is close to.

[0099] After the matching is completed, the area of ​​interest can be determined. For example, if the eye movement data is matched to the segmented area corresponding to the keyboard, the area of ​​interest can be determined according to the position of the keyboard in the video frame. For example, the segmented area corresponding to the keyboard can be determined as the area of ​​interest, or the specific position of the keyboard can be determined according to the outline of the keyboard, and the position can be determined as the area of ​​interest. After the area of ​​interest is determined, the user's gaze behavior in the area of ​​interest can be further analyzed, such as gaze time, gaze count and other indicators, to understand the user's focus and preference on the video content.

[0100] According to the human-factor intelligent dynamic interest area determination method of the present application, the eye movement data of the user watching the video is collected through an eye tracking device; the video frame of the video is segmented by instance to obtain multiple segmented areas of multiple objects on the video frame; the multiple segmented areas are matched according to the position of the eye movement data in the video frame to determine the interest area at the position in the video frame. The embodiment of the present application collects the eye movement data of the user watching the video, and segments the video frame in combination with the instance segmentation technology. By matching the eye movement data with the segmented areas in the video frame, the user's attention can be associated with a specific object or scene in the video, thereby determining the interest area, reducing the tedious process of manual frame-by-frame analysis, and can realize object recognition and tracking across frames and scenes. When the video is played for a long time, the interest area that the user is concerned about can be continuously obtained, thereby realizing dynamic determination of the interest area and improving the efficiency of determining the interest area.

[0101] In some embodiments, performing instance segmentation on a video frame of a video to obtain multiple segmented regions of multiple objects on the video frame includes:

[0102] Identify the pixel boundaries of each object and the category of each object in the video frame;

[0103] The video frame is segmented according to the pixel boundaries of each object to obtain multiple segmented regions of multiple objects on the video frame.

[0104] In this embodiment, a convolutional neural network (CNN) model, such as a Mask R-CNN model, can be used, which can use a region proposal network (RPN) to identify regions that may contain objects, and then use a convolutional neural network to extract features and ultimately determine the pixel boundaries and categories of the objects. After identifying the pixel boundaries of each object, the video frame can be segmented into multiple regions based on these pixel boundaries, each region corresponding to an object.

[0105] In this embodiment, the pixel boundaries of each object in the video frame are identified and the video frame is segmented according to the pixel boundaries of each object, so that the segmentation process has a high correlation with the contour of the object, thereby improving the accuracy of object segmentation.

[0106] In some embodiments, the video frame is segmented according to the pixel boundaries of each object to obtain multiple segmented regions of multiple objects on the video frame, including:

[0107] Mask the pixels within the pixel boundary range of each object to obtain the mask area of ​​each object;

[0108] The mask area surrounding each object is enclosed by the object bounding box;

[0109] The mask region and / or the object bounding box is determined as a plurality of segmented regions of the plurality of objects on the video frame.

[0110] In the field of computer vision and image processing, a mask is a binary image used to identify a specific area or pixel. In this embodiment, the pixels within the pixel boundary range can be masked. For the pixel boundary of each object in the video frame, the pixels within the pixel boundary range can be marked as 1 (or True), and the remaining pixels can be marked as 0 (or False), thereby forming a binary mask image. In the binary mask image, the area where the pixel is marked as 1 is the mask area.

[0111] In this embodiment, an object bounding box may be provided for each mask region of the object to facilitate subsequent processing and visualization. For each mask region, the object bounding box is calculated. For example, the object bounding box may be determined based on the minimum bounding rectangle of the pixels in the mask region.

[0112] In this embodiment, you can choose to use a mask area, an object bounding box, or a combination of the two to represent the segmentation of each object as needed. If you choose a mask area, you can also calculate the area, contour and other feature information of each object mask for further analysis. If you choose an object bounding box, you can record the coordinate information of each bounding box for easy use in visualization.

[0113] In this embodiment, by masking the pixel boundaries of the object, clear object mask regions can be generated, and these mask regions correspond to the actual contours of the object, thereby improving the accuracy of segmentation. Furthermore, by enclosing these mask regions with object bounding boxes, not only can the contours of the object be identified, but also the relative position and spatial range of the object in the video frame can be accurately determined, thereby improving the accuracy of subsequent determination of the region of interest.

[0114] In some embodiments, matching a plurality of segmented regions according to the position of the eye movement data in the video frame to determine the interest region at the position in the video frame includes:

[0115] Matching the position of the eye movement data in the video frame with the position of the mask area of ​​each object in the video frame;

[0116] When the matched eye movement data is within the mask area of ​​the target object, the location of the target object is determined as the area of ​​interest.

[0117] In this embodiment, the matching process may be to compare the position coordinates of the eye movement data in the video frame with the boundary coordinates of the mask area to determine whether the eye movement data falls within the mask area corresponding to a certain object. If the eye movement data is matched to be within the mask area corresponding to a certain object, the object is determined as the target object, and the position of the target object can be determined as the area of ​​interest. For example, the mask area of ​​the target object can be determined as the area of ​​interest, and the object boundary box of the target object can also be determined as the area of ​​interest.

[0118] In this embodiment, by matching the position of the eye movement data with the position of the mask area of ​​each object in the video frame, it is possible to locate the object that the user is actually looking at. When the eye movement data falls within the mask area of ​​an object, the position of the object can be accurately determined as the area of ​​interest, thereby providing more accurate data support for video content analysis and user behavior research in areas such as personalized video content recommendation and advertising placement.

[0119] In some embodiments, matching the position of the eye movement data in the video frame with the position of the mask area of ​​each object in the video frame includes:

[0120] Preliminary matching of the position of the eye movement data in the video frame with the object bounding box of each object;

[0121] When the eye movement data is preliminarily matched to be in the object bounding box of the target object, it is determined whether the eye movement data is in the mask area of ​​the target object.

[0122] In this embodiment, the matching process may be to compare the position coordinates of the eye movement data in the video frame with the boundary coordinates of the object bounding box to determine whether the eye movement data falls within the object bounding box corresponding to a certain object. If the position coordinates of the eye movement data in the video frame fall within the object bounding box corresponding to a certain object, it is necessary to further determine whether the eye movement data is within the mask area of ​​the object.

[0123] In this embodiment, by first using the object bounding box for preliminary matching, the range of objects that the user's line of sight may focus on can be quickly screened out. After the preliminary match is successful, it is further checked whether the eye movement data falls within the mask area of ​​the target object, reducing the area that requires further precise matching, thereby improving matching efficiency.

[0124] In some embodiments, matching a plurality of segmented regions according to the position of the eye movement data in the video frame to determine the interest region at the position in the video frame includes:

[0125] In the case that the matched eye movement data is not in the mask area of ​​any object, the position of the target object corresponding to the mask area closest to the eye movement data is determined as the interest area.

[0126] In this embodiment, if the eye movement data is not within the mask area of ​​any object, the distance between the eye movement data and the mask area of ​​each object can be calculated based on the position coordinates of the eye movement data in the video frame, and the position of the target object corresponding to the mask area closest to the eye movement data is determined as the area of ​​interest.

[0127] In this embodiment, the eye movement data is compared with the object mask area in the video frame, and the mask area closest to the eye movement data is selected if there is no direct match. This method does not require the user's line of sight to fall completely within the object mask area, but allows a certain degree of line of sight deviation, thereby increasing the flexibility of determining the area of ​​interest.

[0128] In some embodiments, matching the position of the eye movement data in the video frame with the position of the mask area of ​​each object in the video frame includes:

[0129] In the case where the eye movement data is matched to the object bounding box of multiple candidate objects,

[0130] The location of the candidate object with the largest object bounding box area is determined as the area of ​​interest;

[0131] Alternatively, different weights are assigned to different candidate objects according to their categories or positions, and the position of the candidate object with the highest weight is determined as the area of ​​interest.

[0132] In this embodiment, if the eye movement data is within the object bounding boxes of multiple candidate objects, in order to determine the area of ​​interest, considering that the user may pay more attention to objects with a large area, the object bounding box corresponding to the large object may contain more important information, the area of ​​the object bounding box of each candidate object can be calculated, and the location of the candidate object with the largest object bounding box area is determined as the area of ​​interest.

[0133] In some embodiments, different weights can be assigned to different objects based on the relevance of the object's category or location. For example, if certain objects are more important than other objects when the user is performing a task, these objects can be given higher weights. For example, in a video frame, objects in a specific location are given a higher weight.

[0134] Let's take an example. Let's say you are analyzing eye tracking data for a web page where a user is looking for specific information on the page. The eye tracking data shows that the user's gaze moves quickly between several links and buttons. By calculating the fixation distance for each link or button, you can determine that the user is most likely to focus on the link or button closest to the fixation point. If the user spends the longest time on a larger button, then that button may represent a larger area of ​​interest. If there is an "OK" button on the web page, you can give that button a higher weight.

[0135] In this embodiment, by selecting the candidate object with the largest object bounding box area as the region of interest, it is considered that the user may pay more attention to objects with large areas, so these areas may contain more important information. Secondly, by assigning weights based on categories or positions to different candidate objects and selecting the object with the highest weight as the region of interest, the user's visual interest can be predicted based on the importance of the object or the relative position to the user's line of sight, thereby improving the accuracy of region of interest recognition.

[0136] In some embodiments, the method further comprises:

[0137] Determine the target object corresponding to the area of ​​interest;

[0138] The position of the interest area in the video frame at the future moment is predicted according to the position of the target object in the video frame at the historical moment and the position of the target object in the current video frame.

[0139] In this embodiment, the target object corresponding to the area of ​​interest can be determined. By tracking the position of the target object in the video frame at a historical moment, the motion trajectory of the object over time can be obtained, and the motion trend of the target object can be analyzed, including parameters such as speed, direction, and acceleration, to predict the position of the target object in the video frame at a future moment.

[0140] In one example, a simple motion model, such as a constant velocity or acceleration model, is used to match a target object in a current frame with a target object in a previous frame to determine the motion trend of the target object, thereby predicting the position of the target object in the next frame.

[0141] In this embodiment, by determining the target object corresponding to the area of ​​interest and using the position of the object in the video frames at historical moments and the current moment, the position of the object at future moments is predicted, thereby optimizing the tracking of the area of ​​interest of the video stream, facilitating continued tracking when the object is obscured or temporarily leaves the field of view, and improving the accuracy of area of ​​interest tracking.

[0142] In some embodiments, the method further comprises:

[0143] Extracting the position information and shape information of the target object corresponding to the area of ​​interest;

[0144] The target object is drawn on the target screen according to the position information and the shape information.

[0145] In this embodiment, the position information and shape information of the object can be extracted from the mask area of ​​the target object. For example, the position information and shape information of the object can be represented by the coordinate information of the boundary pixel points of the object, and these coordinate information can be sent to the target screen. Among them, each object can correspond to a unique identifier, so that the corresponding object can be identified by the unique identifier after each frame in the video is updated, and the latest coordinates of the corresponding object can be synchronously transmitted. In some embodiments, these coordinate information can be transmitted via MQTT (Message Queuing Telemetry Transport).

[0146] In this embodiment, the position and shape of the object in the target screen can be drawn in real time through connecting lines according to the coordinate information obtained in real time, and the specific position and shape of the object in the video can be dynamically displayed, such as Figure 2 shown.

[0147] In this embodiment, by extracting the position and shape information of the target objects corresponding to the area of ​​interest, these objects can be accurately reproduced on the target screen, and the specific performance of the content that the user is concerned about on the screen can be more intuitively seen, and both the position and shape of the objects can be accurately reflected.

[0148] In some embodiments, the method further comprises:

[0149] When the region of interest is updated, the target object drawn on the target screen is removed;

[0150] The target object is drawn on the target screen according to the updated position information and shape information of the target object corresponding to the region of interest.

[0151] In this embodiment, as the video frame in the video is updated, the received coordinate information will also change. In order to make the target screen display the latest updated object, the previously drawn shape on the target screen can be refreshed or removed, and then the target object can be redrawn according to the latest acquired coordinate information, such as Figure 3 shown.

[0152] In this embodiment, by promptly removing the target object drawn on the target screen when the area of ​​interest is updated, and redrawing the object according to the position and shape information of the target object corresponding to the new area of ​​interest, the content displayed on the screen is always the latest information of the object, so that the display of video content is more in line with the user's real-time needs and preferences.

[0153] The human-based intelligent dynamic interest area determination method provided in the embodiment of the present application can be executed by a human-based intelligent dynamic interest area determination device. In the embodiment of the present application, the human-based intelligent dynamic interest area determination method is executed by a human-based intelligent dynamic interest area determination device as an example to illustrate the human-based intelligent dynamic interest area determination device provided in the embodiment of the present application.

[0154] The embodiment of the present application also provides a human-based intelligent dynamic interest area determination device.

[0155] like Figure 4 As shown, the human-based intelligent dynamic interest area determination device includes:

[0156] A collection module 410 collects eye movement data of a user watching a video through an eye tracking device;

[0157] A segmentation module 420 is used to perform instance segmentation on a video frame of the video to obtain multiple segmented regions of multiple objects on the video frame;

[0158] The matching module 430 is used to match the multiple segmented areas according to the position of the eye movement data in the video frame to determine the interest area at the position in the video frame.

[0159] According to the human-factor intelligent dynamic interest area determination device of the present application, the eye movement data of the user watching the video is collected through the eye tracking device; the video frame of the video is segmented by instance to obtain multiple segmented areas of multiple objects on the video frame; the multiple segmented areas are matched according to the position of the eye movement data in the video frame to determine the interest area at the position in the video frame. The embodiment of the present application collects the eye movement data of the user watching the video, and segments the video frame in combination with the instance segmentation technology. By matching the eye movement data with the segmented areas in the video frame, the user's attention can be associated with a specific object or scene in the video, thereby determining the interest area, reducing the tedious process of manual frame-by-frame analysis, and can realize object recognition and tracking across frames and scenes. When the video is played for a long time, the interest area that the user is concerned about can be continuously obtained, thereby realizing dynamic determination of the interest area and improving the efficiency of determining the interest area.

[0160] The human-based intelligent dynamic interest area determination device in the embodiment of the present application can be an edge computing device, or a component in the edge computing device, such as an integrated circuit or a chip. The edge computing device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the edge computing device can be a mobile phone, a tablet computer, a laptop computer, an eye tracker, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application. In an optional example, an edge deployment scheme in which a human-based intelligent dynamic interest area determination device is deployed as an edge computing device or component on the end side can migrate at least part of the computing tasks to the end-side device, giving full play to the advantages of edge computing to improve data processing efficiency, reduce latency and network resource occupancy and dependence, and also facilitate data security and privacy protection.

[0161] The human-based intelligent dynamic interest area determination device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0162] In some embodiments, Figure 5 As shown, an embodiment of the present application also provides an edge computing device 600, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, each process of the above-mentioned human-based intelligent dynamic interest area determination method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0163] It should be noted that the edge computing devices in the embodiments of the present application include the above-mentioned mobile edge computing devices and non-mobile edge computing devices.

[0164] In some embodiments, the edge computing device may be a head-mounted edge computing device; the head-mounted edge computing device may include a camera and an eye tracker.

[0165] An embodiment of the present application also provides a computer-readable storage medium, on which programs or instructions are stored. When the programs or instructions are executed by a processor, the various processes of the above-mentioned human-factor intelligent dynamic interest area determination method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0166] The processor is the processor in the edge computing device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0167] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned human-based intelligent dynamic interest area determination method when executed by a processor.

[0168] The processor is the processor in the edge computing device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0169] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned human-factor intelligent dynamic interest area determination method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0170] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0171] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0172] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0173] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0174] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0175] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for determining a dynamic interest area based on human intelligence, characterized in that: include: Collect eye movement data of users watching videos through eye tracking equipment; Performing instance segmentation on the video frame of the video to obtain multiple segmented regions of multiple objects on the video frame; The multiple segmented areas are matched according to the positions of the eye movement data in the video frame to determine the interest area at the position in the video frame.

2. The method according to claim 1, characterized in that: The performing instance segmentation on the video frame of the video to obtain a plurality of segmented regions of a plurality of objects on the video frame includes: Identifying pixel boundaries of each object and the category of each object in the video frame; The video frame is segmented according to the pixel boundaries of each object to obtain a plurality of segmented regions of a plurality of objects on the video frame.

3. The method according to claim 2, characterized in that The step of segmenting the video frame according to the pixel boundaries of each object to obtain multiple segmented regions of multiple objects on the video frame includes: Mask the pixels within the pixel boundary range of each object to obtain the mask area of ​​each object; The mask area surrounding each object is enclosed by the object bounding box; The mask region and / or the object bounding box is determined as a plurality of segmented regions of a plurality of objects on the video frame.

4. The method according to claim 3, characterized in that The matching of the plurality of segmented regions according to the position of the eye movement data in the video frame to determine the interest region at the position in the video frame includes: Matching the position of the eye movement data in the video frame with the position of the mask area of ​​each object in the video frame; In the case where the eye movement data is matched to be within the mask area of ​​the target object, the location of the target object is determined as the area of ​​interest.

5. The method according to claim 4, wherein the matching of the position of the eye movement data in the video frame with the position of the mask area of ​​each object in the video frame comprises: Preliminarily matching the position of the eye movement data in the video frame with the object bounding box of each object; When it is preliminarily matched that the eye movement data is in the object bounding box of the target object, it is determined whether the eye movement data is in the mask area of ​​the target object.

6. The method according to claim 3, characterized in that The matching of the plurality of segmented regions according to the position of the eye movement data in the video frame to determine the interest region at the position in the video frame includes: In the case that the eye movement data is matched and is not in the mask area of ​​any object, the position of the target object corresponding to the mask area closest to the eye movement data is determined as the area of ​​interest.

7. The method according to claim 3, characterized in that The matching of the position of the eye movement data in the video frame with the position of the mask area of ​​each object in the video frame includes: In the case where the eye movement data is matched to an object bounding box of multiple candidate objects, The location of the candidate object with the largest object bounding box area is determined as the area of ​​interest; Alternatively, different weights are assigned to different candidate objects according to their categories or positions, and the position of the candidate object with the highest weight is determined as the area of ​​interest.

8. The method according to claim 1, characterized in that The method further comprises: Determining a target object corresponding to the area of ​​interest; The position of the interest area in the video frame at a future moment is predicted according to the position of the target object in the video frame at a historical moment and the position of the target object in the current video frame.

9. The method according to claim 1, characterized in that: The method further comprises: Extracting the position information and shape information of the target object corresponding to the area of ​​interest; The target object is drawn on the target screen according to the position information and the shape information.

10. The method according to claim 9, characterized in that The method further comprises: When the area of ​​interest is updated, removing the target object drawn on the target screen; The target object is drawn on the target screen according to the updated position information and shape information of the target object corresponding to the region of interest.

11. The method according to any one of claims 1 to 9, characterized in that: The video includes at least one of the following: Video played on a display device; A video of the user's field of view captured by a camera of a head-mounted edge computing device worn by the user; Video within the user's field of view displayed by a head-mounted edge computing device worn by the user.

12. A human-based intelligent dynamic interest area determination device, characterized in that: include: The acquisition module collects the eye movement data of users watching videos through eye tracking equipment; A segmentation module, used to perform instance segmentation on the video frame of the video to obtain multiple segmented regions of multiple objects on the video frame; A matching module is used to match the multiple segmented areas according to the position of the eye movement data in the video frame to determine the interest area at the position in the video frame.

13. An edge computing device, characterized in that: It includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the human-based intelligent dynamic interest area determination method as described in any one of claims 1 to 11 is implemented.

14. The edge computing device according to claim 13, characterized in that: The edge computing device includes a head-mounted edge computing device; the head-mounted edge computing device includes a camera and an eye tracker.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the method for determining a human-based intelligent dynamic interest area according to any one of claims 1 to 11 is implemented.

Images