Method, apparatus, device and storage medium for information processing
Patent Information
- Application Number
- CN202510442519.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-09
- Publication Date
- 2026-10-16
AI Technical Summary
但是进行图像分割所应用的整体架构过于复杂,因此需要一种更为简单有效的图像分割方法
[0008]It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.
Smart Images

Figure CN122929186A_ABST
Abstract
Description
Technical Field
[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatus, devices and computer-readable storage media for information processing. Background Technology
[0002] Image segmentation is a key technology in computer vision. Its core objective is to divide a digital image into multiple regions or objects with specific meanings, ensuring that pixels within each region exhibit certain consistent characteristics. However, the overall architecture used for image segmentation is overly complex, thus necessitating a simpler and more effective image segmentation method. Summary of the Invention
[0003] In a first aspect of this disclosure, an information processing method is provided. The method includes: receiving a processing request associated with media content; in response to the processing request instructing the segmentation of objects in the media content; acquiring segmentation feature units output by a model based on the media content and the processing request; and generating a segmentation mask for the objects based on the segmentation feature units and a set of visual feature units associated with the media content.
[0004] In a second aspect of this disclosure, an apparatus for information processing is provided. The apparatus includes: a receiving module configured to receive a processing request associated with media content; an acquiring module configured to, in response to a processing request instructing the segmentation of objects in the media content, acquire segmentation feature units output by a model based on the media content and the processing request; and a generating module configured to generate a segmentation mask for the objects based on the segmentation feature units and a set of visual feature units associated with the media content.
[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.
[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.
[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to a first aspect of this disclosure.
[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar embedded elements, wherein:
[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;
[0011] Figure 2 A flowchart illustrating an information processing procedure according to some embodiments of the present disclosure is shown;
[0012] Figure 3 A first schematic diagram showing segmentation using an application model according to some embodiments of the present disclosure is shown;
[0013] Figure 4 A second schematic diagram showing segmentation using an application model according to some embodiments of this disclosure is shown;
[0014] Figure 5 A schematic diagram illustrating segmentation using a baseline model according to some embodiments of the present disclosure is shown;
[0015] Figure 6 A schematic structural block diagram of an apparatus for information processing according to certain embodiments of the present disclosure is shown;
[0016] Figure 7 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation
[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0018] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.
[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0020] The embodiments of this disclosure may involve user data, data acquisition, and / or use. These aspects all comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the user should be informed of the data or data type, scope of use, and usage scenarios that may be involved, and their authorization should be obtained, in accordance with relevant laws and regulations and through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.
[0021] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal data beyond that required for basic functions will not affect the user's use of basic functions.
[0022] Traditional image segmentation methods append a segmentation model to the output of a language model, or embed the language model into the segmentation process. These methods are overly complex, requiring specialized components such as vision-language fusion modules, encoders, and decoders. Furthermore, their final performance often heavily depends on either the language model or the segmentation model; the limitations of individual sub-modules can lead to less than ideal results.
[0023] Embodiments of this disclosure propose an information processing scheme. According to the scheme, a processing request associated with media content is received; in response to the processing request instructing the segmentation of objects in the media content, segmentation feature units output by a model based on the media content and the processing request are obtained; and a segmentation mask for the objects is generated based on the segmentation feature units and a set of visual feature units associated with the media content.
[0024] Based on this approach, the embodiments of this disclosure can implement image segmentation operations using a single model, significantly reducing the complexity of the overall image segmentation architecture. Furthermore, by integrating a learnable upsampling module, the model's segmentation capabilities are fully utilized, resulting in more accurate outputs.
[0025] Example Environment
[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110.
[0027] In some embodiments, electronic device 110 may receive a processing request associated with media content; in response to the processing request instructing the segmentation of objects in the media content, obtain segmentation feature units output by model 120 based on the media content and the processing request; and generate a segmentation mask for the objects based on the segmentation feature units and a set of visual feature units associated with the media content. Model 120 may be deployed on electronic device 110 or on other devices, which will not be elaborated here.
[0028] In some embodiments, the electronic device 110 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 may also support any type of interface for the target user (such as "wearable" circuitry).
[0029] Electronic device 110 can also be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Electronic device 110 may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.
[0030] It should be understood that the structure and function of the various embedded elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.
[0031] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.
[0032] Example process
[0033] Figure 2 A flowchart of an information processing procedure 200 according to some embodiments of the present disclosure is shown. Procedure 200 can be implemented at electronic device 110. Reference is made below. Figure 1 Describe the process 200.
[0034] In box 210, electronic device 110 can receive processing requests associated with media content.
[0035] In some embodiments, media content may include forms of representation such as images and videos. A processing request is used to instruct the segmentation of objects within the media content. For example, if the media content is an image, the received processing request might include "Please segment the cat in the image," in which case the processing request instructs the segmentation of the cat in the image.
[0036] In box 220, electronic device 110 can respond to a processing request instruction to segment objects in the media content and obtain segmentation feature units output by the model based on the media content and the processing request.
[0037] In some embodiments, segmentation feature units are generated by the model based on predefined labels and are used to control model behavior, process specific tasks, or label text structure. Segmentation feature units enhance the model's understanding of input content and its control over output content by explicitly labeling structural or functional information. For example, referencing... Figure 3 The electronic device 110 can respond to the processing request instruction to segment objects in the media content and obtain the segmentation feature unit 307 output by the model 305 based on the media content and the processing request.
[0038] In box 230, electronic device 110 can generate a segmentation mask for an object based on segmentation feature units and a set of visual feature units associated with media content.
[0039] In some embodiments, reference Figure 3 The electronic device 110 can determine the second set of visual feature units 303 to be input into the model based on the media content, and then obtain the first set of visual feature units 306 generated by the model based on the second set of visual feature units 303. For example, if the media content is an image, then the second set of visual feature units 303 corresponding to the image can be determined, and the second set of visual feature units 303 can be input into the model to obtain the first set of visual feature units 306 output by the model.
[0040] In some embodiments, reference Figure 3 The electronic device 110 can divide the media content into multiple image blocks, and then construct a second set of visual feature units 303 based on multiple first embedding representations 301 corresponding to the multiple image blocks. For example, if the media content is an image, the image can be divided into multiple 32×32 image blocks, and then multiple first embedding representations 301 can be determined based on the multiple image blocks, and then a second set of visual feature units 303 can be constructed based on the multiple first embedding representations 301.
[0041] In some embodiments, reference Figure 3 Since the model cannot obtain more detailed feature information when performing image segmentation based solely on multiple image patches, a visual cue information injection mechanism can be used to enable the model to better understand and apply visual cue information, thereby improving the accuracy of the model's segmentation. The electronic device 110 can acquire visual cue information associated with the media content, indicating the region of the reference object within the media content. Then, based on the visual cue information, multiple second embedding representations 302 are constructed. Finally, based on the multiple first embedding representations 301 and the multiple second embedding representations 302, a second set of visual feature units 303 is constructed.
[0042] Visual cues refer to techniques used in computer vision to guide and optimize model performance. By adding specific visual cues to the model input, visual models can better understand and process tasks. In this embodiment, visual cues can be used to indicate the region of a reference object within media content. For example, if the media content is an image containing a cat and a ball, then visual cues can be used to indicate the location of a reference object within the media content. <vp1>The visual cues representing the cat are displayed through... <vp2>This indicates the visual cues associated with the ball.
[0043] Furthermore, visual cues can take many forms, such as dot plots, bounding boxes, and mask images. Visual cues can also be determined based on user interactions with media content. For example, users can determine visual cues by clicking on preset locations within the media content or by adding preset annotations to the media content.
[0044] The second embedding representation 302 is used to fill the corresponding image patch with features corresponding to the visual cue information, and the second embedding representation 302 includes multiple second elements. For example, refer to... Figure 3 The second embedding representation 302 includes multiple second elements marked as empty, marked as 1, and marked as N. The second element marked as 1 is associated with visual cue information. <vp1>Correspondingly, the second element, labeled N, corresponds to the visual cue information. <vpn>Correspondingly, the second element marked as empty has no corresponding visual cue.
[0045] In some embodiments, the first embedding representation 301 includes a plurality of first elements, and there is a positional mapping relationship between the plurality of first elements and the plurality of second elements. Then, a second set of visual feature units 303 injected with visual cue information can be obtained through the fusion operation of the first embedding representation 301 and the second embedding representation 302. This method of early fusion of the first embedding representation 301 and the second embedding representation 302 before inputting them into the model helps the model to understand the visual cue information more effectively.
[0046] For example, if the media content is an image that includes a cat and a ball, and through... <vp1>The visual cues representing the cat are displayed through... <vp2>This represents the visual cue information corresponding to the ball. The second embedded representation 302 includes a second element marked as 1 and a second element marked as 2. The second element marked as 1 corresponds to the visual cue information. <vp1>Correspondingly, the second element marked as 2 corresponds to the visual cue information. <vp2>Correspondingly, the first element corresponding to the second element marked as 1 can be determined by fusing the first embedding representation 301 and the second embedding representation 302, and then combined with the visual cue information. <vp1>Fill in the corresponding cat features; determine the first element corresponding to the second element marked as 2, and match it with the visual cues. <vp2>The corresponding sphere features are then filled in.
[0047] In some embodiments, reference Figure 4 Multiple visual cues can be preset {VP} i |i∈[1,N]}, these visual cues 402 are added to the preset vocabulary corresponding to the model, and a masked visual cue 403 is constructed based on the preset visual cues. Then, the text content 401 included in the processing request is input into the word segmenter 404 so that the text embedding representation VP corresponding to the text content 401 can be applied. t ∈R N×C Visual cues in the form of fill masks Then, feature units corresponding to the visual cues are constructed. In other words, visual feature units V have been added with feature units VP before model processing. N represents the number of visual feature units, C represents the channel size, H and W represent the height and width of the media content, and P represents the size of the image patch.
[0048] In some embodiments, the text content 401 may include visual cues, for example, the text content 401 may be "Please split the image". <vp1>In this way, the model can utilize the visual cues {VP} in the text content 401. i |i∈[1,N]}, accurately identify objects in media content, thereby achieving a better segmentation effect.
[0049] In some embodiments, reference Figure 3 The electronic device 110 can respond to the fact that a target block among multiple image blocks is located within a region, and construct a second embedding representation 302 corresponding to the target block based on the target feature unit corresponding to the visual cue information. The target feature unit indicates the feature corresponding to the visual cue information. For example, if the media content is an image and the image includes a cat, then after dividing the media content into multiple image blocks, some image blocks will fall within the region including the cat, and some image blocks will fall within the region excluding the cat. If the visual cue information corresponds to the region where the cat is located in the image, then the second embedding representation 302 corresponding to the target block can be constructed according to the target feature unit corresponding to the visual cue information. This second embedding representation 302 is used to fill the image block where the cat is located with the features corresponding to the cat.
[0050] In some embodiments, the electronic device 110 may, in response to the target block still being located within the second region corresponding to the second visual cue information, compare the first priority of the first visual cue information with the second priority of the second visual cue information. In response to the first priority being higher than the second priority, a second embedding representation 302 corresponding to the target block is constructed based on the target feature unit corresponding to the visual cue information.
[0051] For example, if the media content is an image depicting a cat playing with a ball, then after dividing the media content into multiple image blocks, there might be target blocks that fall within both the cat's and the ball's regions. We can then compare the first priority of the first cue information corresponding to the cat with the second priority of the second cue information corresponding to the ball. If the first priority is higher than the second priority, we can determine that the target block corresponds to the cat's region. Based on the target feature units corresponding to the visual cue information, we construct a second embedding representation 302 corresponding to the target block. This second embedding representation 302 is used to fill the cat's features into the target block containing the cat.
[0052] In some embodiments, the first priority or the second priority is determined based on the first size of the first region or the second size of the second region. For example, if the target block falls within both the cat's region and the ball's region, the first size of the cat's region and the second size of the ball's region can be determined, and the size of the first and second sizes can be compared to assign a higher priority to the cue information corresponding to the smaller size region. If the second size is smaller than the first size, the region where the ball is located corresponding to the target block can be determined, and a second embedding representation 302 corresponding to the target block can be constructed based on the target feature units corresponding to the visual cue information. This second embedding representation 302 is used to fill the target block containing the features corresponding to the ball.
[0053] In some embodiments, reference Figure 3 The electronic device 110 can use the upsampling unit to upsample a set of visual feature units 306 to determine visual features 309 corresponding to the size of the media content. Then, based on the visual features and segmentation feature units, a segmentation mask for the object is generated. For example, the electronic device 110 can use the upsampling unit to upsample a set of low-resolution visual feature units. S represents the sampling step size, used to determine the high-resolution visual features corresponding to the size of the media content. Then, based on visual features and segmentation feature units, a segmentation mask for the object is generated. In other words, through upsampling units, visual feature units can be... The resolution is increased to four times that of the original media content, which can help a single model fully utilize the capabilities of graphics segmentation.
[0054] In some embodiments, learnable upsampling units can be applied to generate high-resolution visual features. This upsampling unit can efficiently improve the resolution of visual features and optimize feature representation, which is crucial for pixel-level localization. For example, an upsampling unit can include multiple upsampling sub-units, each consisting of a transposed 2D convolution followed by a depthwise separable convolution. A transposed 2D convolution is the inverse of a standard 2D convolution and is commonly used for upsampling and transposing feature maps. A depthwise separable convolution is a convolution operation consisting of two steps: depthwise convolution and pointwise convolution. It can reduce computational cost and parameter count while maintaining the model's expressive power and performance.
[0055] In some embodiments, the model can be trained using distillation loss to improve its fine-grained feature extraction capabilities; that is, the upsampling unit is trained based on the following process: (Refer to...) Figure 3 An upsampling unit can be used to process a set of training visual units 306 to generate a first visual feature 309 corresponding to the training media content. Then, using a pre-trained first segmentation model, based on the first visual feature 309, the first segmentation feature of the training media content is determined. Next, using a pre-trained second segmentation model, based on a set of training visual units 306, the second segmentation feature of the training media content is determined. Finally, based on the comparison between the first and second segmentation features, the upsampling unit is trained.
[0056] Distillation loss is a loss function in knowledge distillation, used to measure the difference between the output of the student model and the output of the expert model, thereby transferring the knowledge of the expert model to the student model. The core idea is to make the student model imitate the behavior of the expert model by matching the output distribution of the two.
[0057] In some embodiments, the distillation loss L can be constructed by comparing the first segmentation feature and the second segmentation feature. distill And predict the loss L through the model ntp Segmentation loss L seg and distillation loss L distill Construct the loss function, L ce For cross-entropy loss, the loss function formula can be as follows:
[0058] L = L ntp +L swg +αL distill
[0059] L seg =λL ce +βL seg
[0060] α is set to 0.5, λ is set to 2.0, and β is set to 0.5. These parameters can also be adjusted according to the specific application.
[0061] In some embodiments, other suitable segmentation models can also be applied during the model training process, without limitation. In this way, expertise in the segmentation domain can be introduced through expert models. This knowledge distillation strategy only results in a negligible increase in the training time of the network model, but it can ensure the optimization of object details and the improvement of segmentation quality without compromising the network model's visual question answering capabilities.
[0062] In some embodiments, processing a request may include text content, which may include visual cues. For example, refer to Figure 3 The media content can be divided into multiple image blocks, and a first embedding representation 301 corresponding to each image block is determined. A second embedding representation 302 is determined based on visual cues, and a text embedding representation 304 is determined based on the text content. The first embedding representation 301 and the second embedding representation 302 are then fused to obtain a second set of visual feature units 303. The second set of visual feature units 303 and the text embedding representation 304 are input into a model 305 to obtain a first set of visual feature units 306 and output text 308, which includes a segmentation feature unit 307. The first set of visual feature units 306 is then upsampled to obtain visual features 309. Finally, visual features 309 and segmentation feature units 307 are multiplied to obtain a segmentation mask for the target object.
[0063] For example, refer to Figure 4 The media content returned a 405 error, and the processing request included "Please describe..." <vp0>Model 408 can output the content in corresponding frame 411: "A gray kitten is sprawled on a wooden table, occupying most of the screen. This cute little guy is holding a knitting needle and a ball of yarn, looking like he's concentrating on knitting..." For example, processing requests may include... Figure 4 The content of 406, "Please segment the smallest ball of yarn," is processed by word segmenter 407, and model 408 can output image 410 accordingly, in which the ball of yarn is marked.
[0064] In some embodiments, a reference can also be constructed. Figure 5 A baseline model can serve as a performance benchmark or reference point, helping researchers evaluate the effectiveness of improvements to the new model. For example, pooling methods can be applied to determine the object representation O∈R based on visual feature units 503. M×C It can be applied Let 503 represent the visual feature unit, M represent the number of visual cue messages, and P represent the size of the image patch. Then, the visual feature unit 503, the embedding representation 504 corresponding to the visual cue messages, and the text embedding representation 505 are input into the model 506 to obtain the visual feature unit 507 and the output text 510, which includes the segmentation feature unit 508.
[0065] Next, the visual feature unit 507 can be reshaped into the visual feature 509 corresponding to the media content, and V∈R can be applied. N×C Represents visual feature unit 507, application Visual feature 509 is represented. Then, visual feature 509 is cross-multiplied with the segmentation feature unit 508 output by the model to generate a segmentation mask. It can be done through Q∈R N×C The segmentation feature unit 508 is represented by K, which represents the number of segmentation feature units.
[0066] The embodiments of this disclosure can implement image segmentation operations using a single model, significantly reducing the complexity of the overall image segmentation architecture. By introducing a learnable upsampling model, low-resolution visual feature units can be optimized within high-resolution visual features. Then, through a visual cue injection mechanism, the model can better understand and apply visual cues, thereby improving the accuracy of the model's segmentation. Furthermore, by employing distillation loss and applying domain-specific expertise in segmentation, the quality of the final generated segmentation mask is improved.
[0067] Example devices and equipment
[0068] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 6 A schematic structural block diagram of an apparatus 600 for information processing according to certain embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in the electronic device 110 discussed above. The various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.
[0069] like Figure 6 As shown, the apparatus 600 includes a receiving module 610 configured to receive a processing request associated with media content; an acquisition module 620 configured to acquire segmentation feature units output by a model based on the media content and the processing request in response to a processing request instruction to segment objects in the media content; and a generation module 630 configured to generate a segmentation mask for the object based on the segmentation feature units and a set of visual feature units associated with the media content.
[0070] In some embodiments, the generation module 630 is configured to upsample a set of visual feature units using an upsampling unit to determine visual features corresponding to the size of the media content; and to generate a segmentation mask for an object based on the visual features and segmentation feature units.
[0071] In some embodiments, the upsampling unit is trained based on the following process: processing a set of training visual units with the upsampling unit to generate a first visual feature corresponding to the training media content; determining a first segmentation feature of the training media content based on the first visual feature using a pre-trained first segmentation model; determining a second segmentation feature of the training media content based on a set of training visual units using a pre-trained second segmentation model; and training the upsampling unit based on a comparison of the first segmentation feature and the second segmentation feature.
[0072] In some embodiments, a set of visual feature units is a first set of visual feature units, and the device 600 may further include a determining module configured to determine a second set of visual feature units to be input to the model based on media content; and a feature unit acquisition module configured to acquire the first set of visual feature units generated by the model based on the second set of visual feature units.
[0073] In some embodiments, the determining module includes a partitioning submodule configured to partition media content into multiple image blocks; and a first construction submodule configured to construct a second set of visual feature units based on multiple first embedding representations corresponding to the multiple image blocks.
[0074] In some embodiments, the first construction submodule includes an acquisition module configured to acquire visual cue information associated with media content, the visual cue information indicating the region of a reference object in the media content; a second construction submodule configured to construct a plurality of second embedding representations based on the visual cue information; and a third construction submodule configured to construct a second set of visual feature units based on the plurality of first embedding representations and the plurality of second embedding representations.
[0075] In some embodiments, the second construction submodule is configured to construct a second embedding representation corresponding to the target block based on the target feature unit corresponding to the visual cue information in response to the target block being located within the region among a plurality of image blocks.
[0076] In some embodiments, the second construction submodule is further configured to, in response to the target block being located within the second region corresponding to the second visual cue information, compare the first priority of the first visual cue information and the second priority of the second visual cue information; and in response to the first priority being higher than the second priority, construct a second embedding representation corresponding to the target block based on the target feature unit corresponding to the visual cue information.
[0077] In some embodiments, the first priority or the second priority is determined based on the first size of the first region or the second size of the second region.
[0078] The units included in device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 600 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.
[0079] Figure 7 A block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to achieve Figure 1 The electronic device 110 shown.
[0080] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.
[0081] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store data and / or data (e.g., training data for training) and can be accessed within electronic device 700.
[0082] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.
[0083] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.
[0084] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).
[0085] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.
[0086] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0087] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable information processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable information processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable information processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0088] Computer-readable program instructions can be loaded onto a computer, other programmable information processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable information processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable information processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0090] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein. < / vpn>
Claims
1. An information processing method, comprising: Receive processing requests associated with media content; In response to the processing request instruction to segment objects in the media content, segmentation feature units output by the model based on the media content and the processing request are obtained; as well as Based on the segmentation feature units and a set of visual feature units associated with the media content, a segmentation mask for the object is generated.
2. The method according to claim 1, wherein, The step of generating a segmentation mask for the object based on the segmentation feature units and a set of visual feature units associated with the media content includes: Upsampling is used to upsample the set of visual feature units to determine the visual features corresponding to the size of the media content; and Based on the visual features and the segmentation feature units, a segmentation mask for the object is generated.
3. The method of claim 2, wherein the upsampling unit is trained based on the following process: The upsampling unit is used to process a set of training visual units to generate a first visual feature corresponding to the training media content; Using a pre-trained first segmentation model, and based on the first visual feature, determine the first segmentation feature of the training media content; Using a pre-trained second segmentation model, based on the set of trained visual units, the second segmentation features of the training media content are determined; as well as The upsampling unit is trained based on the comparison between the first segmentation feature and the second segmentation feature.
4. The method according to claim 1, wherein the set of visual feature units is a first set of visual feature units, further comprising: Based on the media content, a second set of visual feature units to be input into the model is determined; as well as Obtain the first set of visual feature units generated by the model based on the second set of visual feature units.
5. The method according to claim 4, wherein, The step of determining the second set of visual feature units to be input into the model based on the media content includes: The media content is divided into multiple image blocks; and The second set of visual feature units is constructed based on the multiple first embedding representations corresponding to the multiple image blocks.
6. The method according to claim 5, wherein, The construction of the second set of visual feature units based on the multiple first embedding representations corresponding to the multiple image patches includes: Obtain visual cues associated with the media content, the visual cues indicating the area of a reference object within the media content; Based on the visual cue information, multiple second embedding representations are constructed; and The second set of visual feature units is constructed based on the plurality of first embedding representations and the plurality of second embedding representations.
7. The method according to claim 6, wherein, The construction of multiple second embedding representations based on the visual cue information includes: In response to the target block being located within the region among the plurality of image blocks, a second embedding representation corresponding to the target block is constructed based on the target feature unit corresponding to the visual cue information.
8. The method according to claim 7, wherein the visual cue information is first visual cue information, the region is a first region, and the step of constructing the second embedding representation corresponding to the target block based on the target feature unit corresponding to the visual cue information further includes: In response to the target block still being located within the second area corresponding to the second visual cue information, the first priority of the first visual cue information and the second priority of the second visual cue information are compared; as well as In response to the first priority being higher than the second priority, a second embedding representation corresponding to the target block is constructed based on the target feature unit corresponding to the visual cue information.
9. The method of claim 8, wherein the first priority or the second priority is determined based on a first size of the first region or a second size of the second region.
10. An apparatus for information processing, comprising: The receiving module is configured to receive processing requests associated with media content; The acquisition module is configured to acquire segmentation feature units output by the model based on the media content and the processing request in response to the processing request instruction to segment objects in the media content; as well as The generation module is configured to generate a segmentation mask for the object based on the segmentation feature units and a set of visual feature units associated with the media content.
11. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.
12. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 9.
13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.
