A salient object segmentation method, device, equipment and storage medium

Abstract

Embodiments of the present disclosure provide a salient object segmentation method, device and equipment and a storage medium. The method comprises: determining a plurality of gaze point position information when a plurality of users view a target image; determining prompt point position information and a prompt box position information corresponding to a salient object in the target image based on the plurality of gaze point position information; inputting the target image, the prompt point position information and the prompt box position information into a visual basic model to segment the salient object; and obtaining a salient object segmentation result corresponding to the target image based on an output of the visual basic model. Through the technical solution of the embodiments of the present disclosure, the target image can be automatically and accurately segmented, and manual labeling is not required, thereby improving the labeling efficiency and reducing the labeling cost.

Description

Technical Field

[0001] This disclosure relates to computer technology, and more particularly to a salient target segmentation method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of computer technology, neural network models can be used to detect and segment salient objects in images. Currently, manually annotated pixel-level mask images are typically used as sample labels for model training. However, this pixel-level manual annotation method is time-consuming and labor-intensive, reducing annotation efficiency and increasing annotation costs. Summary of the Invention

[0003] This disclosure provides a salient object segmentation method, apparatus, device, and storage medium to automatically and accurately segment images for salient objects without requiring manual annotation, thereby improving annotation efficiency and reducing annotation costs.

[0004] In a first aspect, embodiments of this disclosure provide a salient target segmentation method, including:

[0005] Determine the location information of multiple gaze points when multiple users view a target image;

[0006] Based on multiple gaze point location information, the location information of the cue point and the location information of the cue box corresponding to the salient target in the target image are determined;

[0007] The target image, the location information of the prompt point, and the location information of the prompt box are input into the visual basic model for salient target segmentation;

[0008] Based on the output of the visual basic model, the salient target segmentation result corresponding to the target image is obtained.

[0009] Secondly, embodiments of this disclosure also provide a saliency target segmentation apparatus, comprising:

[0010] The gaze point information determination module is used to determine the position information of multiple gaze points when multiple users view a target image;

[0011] The prompt information determination module is used to determine the prompt point location information and prompt box location information corresponding to the salient target in the target image based on multiple gaze point location information;

[0012] The salient target segmentation module is used to input the target image, the prompt point location information, and the prompt box location information into the visual basic model to segment the salient target;

[0013] The segmentation result determination module is used to obtain the salient target segmentation result corresponding to the target image based on the output of the visual basic model.

[0014] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0015] One or more processors;

[0016] Storage device for storing one or more programs.

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the saliency target segmentation method as described in any embodiment of this disclosure.

[0018] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the saliency target segmentation method as described in any of the embodiments of this disclosure.

[0019] In this embodiment, by using multiple gaze point location information when multiple users view a target image, the location information of cue points and cue boxes corresponding to salient targets in the target image is determined. The target image, cue point location information, and cue box location information are then input into a visual base model. This allows the visual base model to automatically and accurately segment salient targets in the target image using the cue point location information and cue box location information, obtaining the salient target segmentation result corresponding to the target image, i.e., the salient target mask image. Thus, only multiple users need to view the target image, eliminating the need for manual pixel-level annotation, thereby improving annotation efficiency and reducing annotation costs. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 This is a flowchart illustrating a salient target segmentation method provided in an embodiment of this disclosure;

[0022] Figure 2 This is an example of a salient target segmentation process involved in an embodiment of this disclosure;

[0023] Figure 3 This is a flowchart illustrating another salient target segmentation method provided in this embodiment of the disclosure;

[0024] Figure 4 This is an example of a prompt message determination process involved in an embodiment of this disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of a saliency target segmentation device provided in an embodiment of this disclosure;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0027] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0028] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0029] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0031] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0032] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0033] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0034] Figure 1 This is a flowchart illustrating a salient target segmentation method provided in an embodiment of the present disclosure. This embodiment is applicable to the case of performing salient target segmentation on a target image to obtain a salient target mask image. The method can be executed by a salient target segmentation device, which can be implemented in software and / or hardware, or optionally by an electronic device, such as a mobile terminal, a PC, or a server.

[0035] like Figure 1 As shown, the salient target segmentation method specifically includes the following steps:

[0036] S110. Determine the position information of multiple gaze points when multiple users view the target image.

[0037] In this context, the target image refers to the image that requires salient object segmentation. For example, the target image can be a sample image, i.e., the image used during model training. The content of the target image can be arbitrary. A salient object refers to a visually prominent object in the target image. The target image can include one or more salient objects. The gaze position information refers to the image location point where the user's gaze is directed when viewing the target image. Because the viewpoint information of salient objects in the target image is prominent, the user's gaze point is located on the salient object when viewing the target image. Therefore, the gaze position information can be used to measure the position information of salient objects in the target image. There is a one-to-one correspondence between users and gaze position information. Multiple users view the same target image to avoid deviations in the gaze position information when a single user views it, and to ensure that the viewing situation is closer to reality, thus allowing for more accurate localization of the viewed salient object.

[0038] Specifically, multiple users view the same target image, and the gaze position information of each user while viewing the target image is determined. The gaze position information can be determined using any gaze localization method. For example, each user can wear a head-mounted device, such as an eye tracker or VR device. Figure 2 As shown, eye movement information of the user when viewing the target image is obtained through position tracking devices such as eye movement sensors in the head-mounted device, and the eye movement information is converted into pixel coordinates in the image coordinate system, thereby obtaining the gaze point position information of each user when viewing the target image.

[0039] It should be noted that although the gaze point location information of different users when viewing the target image may vary, they will all be within the range of salient targets, such as different positions on the salient target, thus ensuring the accuracy of salient target segmentation. Alternatively, all users can be instructed in advance to deliberately focus on salient targets in the target image to collect more accurate gaze point location information, further improving the accuracy of salient target segmentation.

[0040] For example, S110 may include: acquiring eye movement information of multiple users watching the target video; and determining multiple gaze point position information corresponding to each target image in the target video based on the eye movement information.

[0041] In this context, the target video refers to the video for which salient object segmentation is required for each video frame. Each video frame in the target video represents a target image. Specifically, multiple users watch the same target video, and the eye movement information of each user while watching the target video is determined. This eye movement information is then transformed into coordinates to determine the gaze position information of each user for each target image in the target video. For example, each user can wear a head-mounted device, such as an eye tracker or VR device. The eye movement information of the user while watching the target video is obtained through position tracking devices such as eye sensors in the head-mounted device, and the gaze position information of each user for each target image in the target video is determined based on the eye movement information. Since the gaze position information is obtained during the user's continuous viewing of the target video, the user's gaze can be more focused on the salient object, thus obtaining more accurate gaze position information. This enables salient object segmentation and annotation at the video granularity, and further improves the accuracy of salient object segmentation and annotation by considering the temporal characteristics of continuous video playback.

[0042] S120. Based on the position information of multiple gaze points, determine the position information of the cue point and the position information of the cue box corresponding to the salient target in the target image.

[0043] The cue point location information can be the location information of the center point used to indicate the salient target. The cue box location information can be the location information of the bounding box used to indicate the salient target. Each salient target in the target image corresponds to one cue point location information and one cue box location information.

[0044] Specifically, such as Figure 2As shown, a gaze point encoder can be used to convert all gaze point location information into cue point location information and cue box location information corresponding to each salient object in the target image. The cue point location information helps determine the location of the salient object, and the cue box location information helps distinguish the foreground and background of the image. Therefore, by combining a cue point location information and a cue box location information, a salient object in the target image can be uniquely and accurately identified.

[0045] For example, S120 may include: performing gaze point clustering based on multiple gaze point location information to obtain a clustered gaze point set and center point location information corresponding to the gaze point set; determining cue point location information corresponding to a salient target in the target image based on the center point location information corresponding to the gaze point set; and determining cue box location information corresponding to a salient target in the target image based on the gaze point location information in the gaze point set.

[0046] Specifically, any clustering method can be used to cluster all gaze point location information. Each gaze point set obtained after clustering corresponds to a salient target; that is, the number of gaze point sets is equal to the number of salient targets in the target image. For example, k-means clustering can be used to cluster all gaze point location information into N classes (i.e., N clusters, the target image contains N salient targets), thereby obtaining the gaze point set corresponding to each salient target. Based on the gaze point location information in each gaze point set, the corresponding center point location information is obtained, i.e., the center point location information of each salient target. For each salient target in the target image, i.e., each gaze point set, the center point location information corresponding to that gaze point set can be determined as the cue point location information corresponding to the salient target. Based on the distribution range of the gaze point location information in that gaze point set, the edge contour of the salient target can be determined, thereby obtaining the cue box location information corresponding to the salient target.

[0047] S130. Input the target image, cue point location information, and cue box location information into the visual base model to segment the salient target.

[0048] The visual foundation model refers to a basic model with image segmentation capabilities, pre-trained on a large-scale dataset. It can be a pre-trained interactive segmentation model that segments objects based on prompts. For example, it could be a large-scale visual model like SAM (Segment Anything Model) or SEEM (Segment Everything Everywhere All at Once). Pre-trained on extensive labeled data, the visual foundation model exhibits good generalization and robustness, enabling it to adapt to downstream tasks in various niche scenarios.

[0049] Specifically, the target image, the location information of the cue points corresponding to each salient object in the target image, and the location information of the cue boxes can be input into a pre-trained visual base model. This allows the visual base model to accurately determine the location of each salient object in the target image and perform corresponding segmentation based on the location information of the cue points and cue boxes corresponding to each salient object, and then output the segmented salient object segmentation results. For example... Figure 2 As shown, the visual foundation model can include an image encoder, a cue encoder, and a mask decoder. The specific segmentation process in the visual foundation model is as follows: The target image is input into the image encoder. The cue point location information and cue box location information corresponding to each salient object in the target image are input into the cue encoder. In the image encoder, the input target image is encoded into image vector information in a high-dimensional feature space. In the cue encoder, the input cue point location information and cue box location information are encoded into corresponding cue vector information. The image vector information and cue vector information are input into the mask decoder for decoding, determining the location of each salient object in the target image for segmentation, and outputting the segmented salient object segmentation result.

[0050] S140. Based on the output of the visual basic model, obtain the salient target segmentation result corresponding to the target image.

[0051] The salient object segmentation result can refer to the matting of salient objects in the target image, or it can refer to a salient object mask image with the same size as the target image. Specifically, the visual base model can directly output the salient object segmentation result with the highest segmentation quality score, or it can output multiple segmentation results and their corresponding segmentation quality scores, and use the segmentation result with the highest segmentation quality score as the final salient object segmentation result for the target image. The segmentation quality score can be used to characterize the completeness and edge regularity of the segmentation result, etc.

[0052] It is important to note that this embodiment only requires multiple users to view the target image, eliminating the need for manual user operation. Compared to traditional mouse operation, it is more convenient and lightweight, reducing the manual cost of target segmentation. Furthermore, by determining the composite cue point location information and cue box location information based on the gaze point location information, the results segmented using the viewpoint-based model have stronger subjective relevance, enabling more accurate segmentation of salient targets in complex scenes, thereby improving the accuracy and applicability of salient target segmentation.

[0053] The technical solution of this disclosure determines the cue point location information and cue box location information corresponding to salient targets in the target image based on the location information of multiple gaze points when multiple users view the target image. The target image, cue point location information, and cue box location information are then input into a visual base model. This allows the visual base model to automatically and accurately segment the salient targets in the target image using the cue point location information and cue box location information, obtaining the salient target segmentation result corresponding to the target image, i.e., the salient target mask image. As a result, only multiple users need to view the target image, eliminating the need for manual pixel-level annotation, thereby improving annotation efficiency and reducing annotation costs.

[0054] Based on the above technical solution, after S140, it may also include: training the salient target segmentation model based on the target image and the corresponding salient target segmentation results.

[0055] The salient object segmentation model can be any neural network model used to detect salient objects in a segmented image. For example, it can be a network model used in the Salient Object Detection (SOD) task. Before use, the salient object segmentation model needs to be trained to ensure its accuracy. Specifically, the target image can be used as the sample image, and the corresponding salient object segmentation results as the sample labels. Supervised training of the salient object segmentation model can be performed, utilizing the accurate sample labels quickly obtained in steps S110-S140 to improve the training effect.

[0056] Figure 3 This is a flowchart illustrating another salient target segmentation method provided in this embodiment. Based on the above-described embodiments, this embodiment optimizes the step of "determining the location information of the cue box corresponding to the salient target in the target image based on the location information of the gaze points in the gaze point set." Explanations of terms that are the same as or corresponding to those in the above-described embodiments are not repeated here.

[0057] like Figure 3As shown, the salient target segmentation method specifically includes the following steps:

[0058] S310. Determine the position information of multiple gaze points when multiple users view the target image.

[0059] S320. Based on multiple viewpoint location information, perform gaze point clustering to obtain the clustered gaze point set and the center point location information corresponding to the gaze point set.

[0060] Specifically, gaze point clustering can be performed using any clustering method based on multiple gaze point location information. All gaze points are clustered into multiple categories, each corresponding to a salient object and a set of gaze points. The number of gaze point sets equals the number of salient objects in the target image. For example, k-means clustering can be used to cluster all gaze point location information into N classes (i.e., N clusters, meaning the target image contains N salient objects), thereby obtaining the gaze point set corresponding to each salient object and the center point location information of each cluster.

[0061] S330. Based on the center point location information corresponding to the gaze point set, determine the cue point location information corresponding to the salient target in the target image.

[0062] Specifically, for each salient target in the target image, i.e., each set of gaze points, the center point location information corresponding to that set of gaze points can be determined as the cue point location information corresponding to that salient target. For example... Figure 4 As shown, there are three salient targets in the target image: a car, a cat, and a racket. Accordingly, all gaze points are clustered into three classes, i.e., three sets of gaze points, and the location information of the center point of each class is determined as the location information of the cue point corresponding to the salient target.

[0063] S340. Based on the gaze point location information in the gaze point set, perform Gaussian blur processing on the gaze points to generate a thermal image.

[0064] A heat map, in this context, refers to a grayscale image where pixel values ​​range from 0 to 255. The pixel values ​​in a heat map represent the probability that a pixel is located within a salient target area. For example, if a pixel is not within a salient target area, its corresponding pixel value is 0, meaning that the pixel will appear black in the heat map.

[0065] Specifically, for each set of gaze points, a thermal image can be generated by applying Gaussian blurring to each gaze point using a preset Gaussian kernel radius, based on the location information of each gaze point in the set. The preset Gaussian kernel radius can refer to the radius of the visible area corresponding to each gaze point, which can be determined in advance based on the distance between the user and the viewing screen and the viewing angle.

[0066] For example, S340 may include: performing binarization processing on the gaze points based on the gaze point location information in the gaze point set to obtain a gaze point image; and performing Gaussian blur processing on each gaze point in the gaze point image to generate a thermal image.

[0067] The fixation point image is a binary image, meaning the pixel value is either 0 or 255. The pixel value at the fixation point location in the fixation point image is 255, while the pixel value at other locations is 0.

[0068] Specifically, for each set of gaze points, a point with a preset radius and a pixel value of 255 can be drawn at the location information of each gaze point in the set to represent the gaze point, thereby obtaining a gaze point image, for example... Figure 4 Each white dot in the mid-focal point image represents a focal point. A Gaussian blur is applied to each focal point in the set using a preset Gaussian kernel radius; that is, each focal point is represented by a white blurred region with a preset Gaussian kernel radius, thus obtaining a thermal image.

[0069] S350. Based on the heat map, determine the simply connected regions in the heat map.

[0070] Specifically, based on the properties of Gaussian blur, thermal images contain simply connected regions, which can be identified within the thermal image. These simply connected regions can be used to characterize the contour regions of salient targets.

[0071] For example, S350 may include: performing binarization processing on the heat image to generate a binary image corresponding to the heat image; and determining a simply connected region in the binary image.

[0072] Specifically, such as Figure 4 As shown, pixel thresholding can be used to binarize the thermal image. Pixel values ​​greater than or equal to the threshold are set to 255, and pixel values ​​less than the threshold are set to 0. This makes salient target areas appear white and background areas appear black, thus obtaining a binary image that clearly distinguishes the target and background areas. By identifying simply connected regions in the binary image, the contour regions of salient targets can be obtained more accurately, further improving the accuracy of salient target segmentation.

[0073] S360. Based on a simply connected region, determine the location information of the prompt box corresponding to the salient target in the target image.

[0074] Specifically, the location information of the cue box corresponding to each salient target in the target image can be determined based on the location information of the simply connected region.

[0075] For example, S340 may include: determining the bounding box of a simply connected region; and determining the location information of the cue box corresponding to a salient target in the target image based on the bounding box.

[0076] Specifically, the bounding box of a single connected region can be determined based on a preset box shape, such as determining the bounding rectangle of the single connected region, for example... Figure 4 As shown, the diagonal position information of the bounding box, such as the position information of the upper left and upper right corners, can be determined as the position information of the tooltip corresponding to the salient target, so that the overall outline area of ​​the salient target can be located based on the position information of the tooltip.

[0077] S370. Input the target image, cue point location information, and cue box location information into the visual basic model to segment salient targets.

[0078] S380. Based on the output of the visual basic model, obtain the salient target segmentation result corresponding to the target image.

[0079] For example, such as Figure 4 As shown, the target image, the location information of the cue points and the location information of the cue boxes corresponding to each salient target in the target image are input into the visual base model. Based on the input cue point location information and cue box location information, the visual base model segments each salient target in the input target image and outputs three salient target mask images, thereby achieving accurate segmentation of salient targets.

[0080] The technical solution of this disclosure generates a thermal image by performing Gaussian blurring on the gaze point location information based on the gaze point location information in the gaze point set, and accurately determines the location information of the prompt box corresponding to the salient target in the target image based on the simply connected region in the thermal image, thereby further improving the accuracy of salient target segmentation.

[0081] Figure 5 This is a schematic diagram of the structure of a saliency target segmentation device provided in an embodiment of the present disclosure, as shown below. Figure 5 As shown, the device specifically includes: a gaze point information determination module 510, a prompt information determination module 520, a salient target segmentation module 530, and a segmentation result determination module 540.

[0082] The system includes a gaze point information determination module 510, used to determine the position information of multiple gaze points when multiple users view a target image; a prompt information determination module 520, used to determine the position information of prompt points and prompt boxes corresponding to salient targets in the target image based on the multiple gaze point position information; a salient target segmentation module 530, used to input the target image, the prompt point position information, and the prompt box position information into a visual basic model for salient target segmentation; and a segmentation result determination module 540, used to obtain the salient target segmentation result corresponding to the target image based on the output of the visual basic model.

[0083] The technical solution provided in this disclosure determines the cue point location information and cue box location information corresponding to salient targets in the target image based on the location information of multiple gaze points when multiple users view the target image. The target image, cue point location information, and cue box location information are then input into a visual basic model. This allows the visual basic model to automatically and accurately segment the salient targets in the target image using the cue point location information and cue box location information, obtaining the salient target segmentation result corresponding to the target image, i.e., the salient target mask image. As a result, only multiple users need to view the target image, eliminating the need for manual pixel-level annotation, thereby improving annotation efficiency and reducing annotation costs.

[0084] Based on the above technical solution, the gaze point information determination module 510 is specifically used to: acquire eye movement information of multiple users watching a target video; and determine multiple gaze point position information corresponding to each target image in the target video based on the eye movement information.

[0085] Based on the above technical solutions, the prompt information determination module 520 includes:

[0086] The gaze point clustering submodule is used to cluster gaze points based on multiple gaze point location information to obtain a clustered set of gaze points and the center point location information corresponding to the set of gaze points.

[0087] The cue point information determination submodule is used to determine the cue point location information corresponding to the salient target in the target image based on the center point location information corresponding to the gaze point set;

[0088] The prompt box information determination submodule is used to determine the prompt box position information corresponding to the salient target in the target image based on the gaze point position information in the gaze point set.

[0089] Based on the above technical solutions, the prompt box information determination submodule includes:

[0090] The thermal image generation unit is used to perform Gaussian blurring of the gaze points based on the gaze point location information in the gaze point set to generate a thermal image.

[0091] A single connected region determination unit is used to determine single connected regions in the heat image based on the heat image;

[0092] The prompt box information determination unit is used to determine the prompt box position information corresponding to the salient target in the target image based on the single connected region.

[0093] Based on the above technical solutions, the heat image generation unit is specifically used for:

[0094] Based on the gaze point location information in the gaze point set, gaze point binarization is performed to obtain a gaze point image; Gaussian blurring is then applied to each gaze point in the gaze point image to generate a thermal image.

[0095] Based on the above technical solutions, the single-connected region determination unit is specifically used for:

[0096] The heat image is binarized to generate a corresponding binary image; the simply connected regions in the binary image are then identified.

[0097] Based on the above technical solutions, the prompt box information determination unit is specifically used for:

[0098] Determine the bounding box of the simply connected region; based on the bounding box, determine the location information of the cue box corresponding to the salient target in the target image.

[0099] Based on the above technical solutions, the device also includes:

[0100] The model training module is used to train a salient object segmentation model based on the target image and the salient object segmentation result corresponding to the target image after obtaining the salient object segmentation result corresponding to the target image.

[0101] The salient target segmentation apparatus provided in this disclosure can execute the salient target segmentation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the salient target segmentation method.

[0102] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0103] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 6 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 6 The diagram below shows the structure of the terminal device or server 500. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0104] like Figure 6 As shown, electronic device 500 may include a processing unit (e.g., central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An edit / output (I / O) interface 505 is also connected to bus 504.

[0105] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0106] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0107] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0108] The electronic device provided in this embodiment belongs to the same inventive concept as the salient target segmentation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0109] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the saliency target segmentation method provided in the above embodiments.

[0110] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0111] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0112] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0113] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine multiple gaze point location information when multiple users view a target image; determine cue point location information and cue box location information corresponding to salient objects in the target image based on the multiple gaze point location information; input the target image, the cue point location information, and the cue box location information into a visual basic model for salient object segmentation; and obtain the salient object segmentation result corresponding to the target image based on the output of the visual basic model.

[0114] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0116] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0117] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0118] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] According to one or more embodiments of this disclosure, [Example 1] provides a salient target segmentation method, including:

[0120] Determine the location information of multiple gaze points when multiple users view a target image;

[0121] Based on multiple gaze point location information, the location information of the cue point and the location information of the cue box corresponding to the salient target in the target image are determined;

[0122] The target image, the location information of the prompt point, and the location information of the prompt box are input into the visual basic model for salient target segmentation;

[0123] Based on the output of the visual basic model, the salient target segmentation result corresponding to the target image is obtained.

[0124] According to one or more embodiments of this disclosure, [Example 2] provides a salient target segmentation method, which further includes:

[0125] Optionally, determining the location information of multiple gaze points when multiple users view the target image includes:

[0126] Acquire eye-tracking information from multiple users while they watch a target video;

[0127] Based on the eye-tracking information, the position information of multiple gaze points corresponding to each target image in the target video is determined.

[0128] According to one or more embodiments of this disclosure, [Example 3] provides a salient target segmentation method, which further includes:

[0129] Optionally, determining the cue point location information and cue box location information corresponding to the salient target in the target image based on multiple gaze point location information includes:

[0130] Based on multiple gaze point location information, gaze point clustering is performed to obtain a clustered gaze point set and the center point location information corresponding to the gaze point set;

[0131] Based on the center point location information corresponding to the set of gaze points, determine the cue point location information corresponding to the salient target in the target image;

[0132] Based on the gaze point location information in the gaze point set, the location information of the cue box corresponding to the salient target in the target image is determined.

[0133] According to one or more embodiments of this disclosure, [Example 4] provides a salient target segmentation method, which further includes:

[0134] Optionally, determining the location information of the cue box corresponding to the salient target in the target image based on the location information of the gaze points in the gaze point set includes:

[0135] Gaussian blurring of the gaze points is performed on the gaze point location information in the gaze point set to generate a thermal image;

[0136] Based on the heat map, identify the simply connected regions in the heat map;

[0137] Based on the simply connected region, determine the location information of the prompt box corresponding to the salient target in the target image.

[0138] According to one or more embodiments of this disclosure, [Example 5] provides a salient target segmentation method, which further includes:

[0139] Optionally, the step of performing Gaussian blur processing on the gaze points based on the gaze point location information in the gaze point set to generate a thermal image includes:

[0140] Binarization of the gaze points is performed based on the gaze point location information in the gaze point set to obtain a gaze point image;

[0141] Gaussian blurring is applied to each gaze point in the gaze point image to generate a thermal image.

[0142] According to one or more embodiments of this disclosure, [Example Six] provides a salient target segmentation method, further comprising:

[0143] Optionally, determining the simply connected regions in the heatmap based on the heatmap includes:

[0144] The heat image is binarized to generate a corresponding binary image.

[0145] Identify the simply connected regions in the binary image.

[0146] According to one or more embodiments of this disclosure, [Example Seven] provides a salient target segmentation method, further comprising:

[0147] Optionally, determining the location information of the cue box corresponding to the salient target in the target image based on the simply connected region includes:

[0148] Determine the bounding box of the single connected region;

[0149] Based on the bounding box, determine the location information of the prompt box corresponding to the salient target in the target image.

[0150] According to one or more embodiments of this disclosure, [Example Eight] provides a salient target segmentation method, further comprising:

[0151] Optionally, after obtaining the salient target segmentation result corresponding to the target image, the method further includes:

[0152] Based on the target image and the corresponding salient target segmentation results, the salient target segmentation model is trained.

[0153] According to one or more embodiments of this disclosure, [Example Nine] provides a saliency target segmentation apparatus, comprising:

[0154] The gaze point information determination module is used to determine the position information of multiple gaze points when multiple users view a target image;

[0155] The prompt information determination module is used to determine the prompt point location information and prompt box location information corresponding to the salient target in the target image based on multiple gaze point location information;

[0156] The salient target segmentation module is used to input the target image, the prompt point location information, and the prompt box location information into the visual basic model to segment the salient target;

[0157] The segmentation result determination module is used to obtain the salient target segmentation result corresponding to the target image based on the output of the visual basic model.

[0158] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0159] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0160] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A salient target segmentation method, characterized in that, include: Acquire eye-tracking information from multiple users while they watch a target video; Based on the eye-tracking information, determine the position information of multiple fixation points corresponding to each target image in the target video; The target image is each video frame of the target video; Based on the multiple gaze point location information, gaze point clustering is performed to obtain the clustered gaze point set and the center point location information corresponding to the gaze point set, wherein the number of gaze point sets is equal to the number of salient targets in the target image; Based on the center point location information corresponding to the set of gaze points, the cue point location information corresponding to the salient target in the target image is determined, and the cue point location information is used to indicate the center point location information of the salient target; Based on the gaze point location information in the gaze point set, Gaussian blurring is performed on each gaze point using a preset Gaussian kernel radius to generate a thermal image; the pixel values ​​in the thermal image are used to characterize the probability value of a pixel being located within the salient target range; the preset Gaussian kernel radius refers to the radius of the visible area corresponding to each gaze point, and the preset Gaussian kernel radius is determined in advance based on the distance between the user and the viewing screen and the viewing angle; Based on the heat map, identify the simply connected regions in the heat map; Based on the simply connected region, the location information of the prompt box corresponding to the salient target in the target image is determined, and the location information of the prompt box is used to prompt the location information of the edge box of the salient target; The target image, the location information of the prompt point, and the location information of the prompt box are input into the visual basic model for salient target segmentation; Based on the output of the visual basic model, the salient target segmentation result corresponding to the target image is obtained.

2. The salient target segmentation method according to claim 1, characterized in that, The step of generating a thermal image by performing Gaussian blurring on each gaze point using a preset Gaussian kernel radius based on the gaze point location information in the gaze point set includes: Binarization of the gaze points is performed based on the gaze point location information in the gaze point set to obtain a gaze point image; A thermal image is generated by applying Gaussian blur to each gaze point in the gaze point image using a preset Gaussian kernel radius.

3. The salient target segmentation method according to claim 1, characterized in that, The step of determining simply connected regions in the heatmap based on the heatmap includes: The heat image is binarized to generate a corresponding binary image. Identify the simply connected regions in the binary image.

4. The salient target segmentation method according to claim 1, characterized in that, The step of determining the location information of the cue box corresponding to the salient target in the target image based on the simply connected region includes: Determine the bounding box of the single connected region; Based on the bounding box, determine the location information of the prompt box corresponding to the salient target in the target image.

5. The salient target segmentation method according to any one of claims 1-4, characterized in that, After obtaining the salient target segmentation result corresponding to the target image, the method further includes: Based on the target image and the corresponding salient target segmentation results, the salient target segmentation model is trained.

6. A salient target segmentation device, characterized in that, include: The fixation point information determination module is used to acquire eye movement information of multiple users watching the target video; Based on the eye-tracking information, determine the position information of multiple fixation points corresponding to each target image in the target video; The target image is each video frame of the target video; The prompt information determination module is used to perform gaze point clustering based on the multiple gaze point position information to obtain a clustered gaze point set and the center point position information corresponding to the gaze point set, wherein the number of gaze point sets is equal to the number of salient objects in the target image; based on the center point position information corresponding to the gaze point set, determine the prompt point position information corresponding to the salient object in the target image, the prompt point position information is used to indicate the center point position information of the salient object; based on the gaze point position information in the gaze point set, perform Gaussian blur processing on each gaze point using a preset Gaussian kernel radius to generate a thermal image; the pixel values ​​in the thermal image are used to characterize the probability value of the pixel being located within the range of the salient object; the preset Gaussian kernel radius refers to the radius of the visible area corresponding to each gaze point, the preset Gaussian kernel radius is predetermined based on the distance between the user and the viewing screen and the viewing angle; based on the thermal image, determine the simply connected regions in the thermal image; based on the simply connected regions, determine the prompt box position information corresponding to the salient object in the target image, the prompt box position information is used to indicate the edge box position information of the salient object; The salient target segmentation module is used to input the target image, the prompt point location information, and the prompt box location information into the visual basic model to segment the salient target; The segmentation result determination module is used to obtain the salient target segmentation result corresponding to the target image based on the output of the visual basic model.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the salient target segmentation method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The system stores computer instructions that, when executed by a computer processor, are used to perform the salient target segmentation method as described in any one of claims 1-5.

Images