Image instance segmentation method, storage medium and electronic device

Abstract

The application discloses an image instance segmentation method, a storage medium and an electronic device. The method comprises the following steps: acquiring a to-be-processed image and a to-be-processed query request, wherein the display content in the to-be-processed image is a target object, the to-be-processed query request comprises a position embedding part and a content part, the position embedding part is used for predicting position information of the target object, and the content part is used for target detection on the target object; and performing camouflage instance segmentation on the to-be-processed image based on the to-be-processed query request to obtain a segmentation result corresponding to the target object. The application solves the technical problems of low efficiency, low segmentation result accuracy and large resource consumption in the instance segmentation process caused by the instance segmentation method with a large number of redundant queries in the related art.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of artificial intelligence technology. Specifically, it relates to an image instance segmentation method, a storage medium, and an electronic device. Background Technology

[0002] In the field of computer science, particularly artificial intelligence, image instance segmentation methods are increasingly widely used. Among related technologies, query-based instance segmentation architectures typically use a large number of queries to aggregate semantic information of instances in the training dataset. However, since there is a one-to-one correspondence between queries and instance objects, usually only a few object instances in the image corresponding to a query are activated. A large number of queries are highly redundant for a single image, leading to false positives (i.e., all queries are effective for instances in the entire dataset). This results in low efficiency, high resource consumption, and low accuracy in the image instance segmentation process.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides an image instance segmentation method, storage medium, and electronic device to at least solve the technical problems in related technologies where instance segmentation methods with a large number of redundant queries result in low efficiency, low accuracy of segmentation results, and high resource consumption.

[0005] According to one aspect of the present invention, an image instance segmentation method is provided, comprising: acquiring an image to be processed and a query request to be processed, wherein the display content in the image to be processed is a target object, and the query request to be processed includes a location embedding part and a content part, wherein the location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object; and performing camouflaged instance segmentation on the image to be processed based on the query request to be processed to obtain a segmentation result corresponding to the target object.

[0006] According to another aspect of the present invention, an image instance segmentation method is also provided, comprising: acquiring an image of a city building complex to be processed and a query request to be processed, wherein the display content in the image of the city building complex to be processed is a target building, and the query request to be processed includes a location embedding part and a content part, wherein the location embedding part is used to predict the location information of the target building, and the content part is used to perform target detection on the target building; and performing camouflage instance segmentation on the image of the city building complex to be processed based on the query request to be processed to obtain a segmentation result corresponding to the target building.

[0007] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any of the above-described image instance segmentation methods.

[0008] According to another aspect of the present invention, an electronic device is also provided, including: a processor; and a memory connected to the processor, configured to provide the processor with instructions for processing the following processing steps: acquiring an image to be processed and a query request to be processed, wherein the display content in the image to be processed is a target object, and the query request to be processed includes: a location embedding part and a content part, the location embedding part being used to predict the location information of the target object, and the content part being used to perform target detection on the target object; and performing camouflage instance segmentation on the image to be processed based on the query request to be processed to obtain a segmentation result corresponding to the target object.

[0009] In this embodiment of the invention, an image to be processed and a query request to be processed are obtained. The image to be processed contains the target object, and the query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object. Furthermore, based on the query request to be processed, the image to be processed is segmented into a disguised instance to obtain the segmentation result corresponding to the target object.

[0010] It is noteworthy that, through the embodiments of the present invention, by considering the location embedding part and content part involved in the query request during the query-based instance segmentation process, the image to be processed containing the target object is subjected to spoofed instance segmentation to obtain the segmentation result corresponding to the target object. This achieves the purpose of considering the content and location embedding corresponding to the query to perform query-based spoofed instance segmentation of the target object, thereby achieving the technical effect of improving the efficiency and segmentation accuracy of the query-based spoofed instance segmentation process while reducing resource consumption. This solves the technical problem in related technologies that the instance segmentation method with a large number of redundant queries results in low efficiency, low accuracy of segmentation results and high resource consumption. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0012] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing an image instance segmentation method is shown.

[0013] Figure 2This is a flowchart of an image instance segmentation method according to an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of an optional adaptive query architecture according to an embodiment of this application;

[0015] Figure 4 This is a flowchart of another image instance segmentation method according to an embodiment of this application;

[0016] Figure 5 This is a schematic diagram of the structure of an image instance segmentation device according to an embodiment of this application;

[0017] Figure 6 This is a schematic diagram of an optional image instance segmentation device according to an embodiment of this application;

[0018] Figure 7 This is a schematic diagram of another optional image instance segmentation device according to an embodiment of this application;

[0019] Figure 8 This is a schematic diagram of another image instance segmentation device according to an embodiment of this application;

[0020] Figure 9 This is a schematic diagram of another optional image instance segmentation device according to an embodiment of this application;

[0021] Figure 10 This is a structural block diagram of another computer terminal according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0025] Camouflaged Instance Segmentation (CIS) refers to the process of segmenting camouflaged instances in an image that have a high intrinsic similarity to the background. Compared with semantic segmentation, instance segmentation can identify different individuals of the same object.

[0026] Example 1

[0027] According to an embodiment of this application, an embodiment of an image instance segmentation method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] The method embodiment provided in Embodiment 1 of this application can be executed in a mobile terminal, computer terminal or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an image instance segmentation method is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor (MCU) or a field-programmable gate array (FPGA) or similar processing device), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a computer bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0029] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0030] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image instance segmentation method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the image instance segmentation method described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0031] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0032] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0033] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance and is intended to illustrate the types of components that may exist in the aforementioned computer device (or mobile device).

[0034] Under the aforementioned operating environment, this application provides the following: Figure 2 This illustrates an image instance segmentation method. Figure 2 This is a flowchart of an image instance segmentation method according to an embodiment of this application, such as... Figure 2 As shown, the image instance segmentation method includes:

[0035] Step S21: Obtain the image to be processed and the query request to be processed. The display content in the image to be processed is the target object. The query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object.

[0036] Step S22: Based on the query request to be processed, perform spoofed instance segmentation on the image to be processed to obtain the segmentation result corresponding to the target object.

[0037] In the above embodiments of this application, the target object in the image to be processed is the segmentation target for camouflage instance segmentation. The query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object. That is, according to the method provided by the embodiments of this application, in the query-based camouflage instance segmentation process, a query paradigm that considers both location embedding and query content is adopted to perform camouflage instance segmentation on the image to be processed, thereby obtaining the segmentation result corresponding to the target object.

[0038] It is readily understood that in the field of computer technology, especially in the field of artificial intelligence, existing technologies for spoofed instance segmentation typically employ a two-stage spoofed instance segmentation model. However, unlike existing technologies, the image instance segmentation method provided in this application uses a single-stage algorithm for spoofed instance segmentation, utilizing a query paradigm for concise spoofed instance segmentation. Specifically, in the image instance segmentation method provided in this application, each query according to the query paradigm includes two functions: content clustering and location embedding.

[0039] Specifically, assuming each query corresponds to an object instance in an image, a predefined set of external object queries maps the content and location information of each query to the corresponding object instance in the image. During this process, a large number of fixed queries are used to aggregate the semantic information of the instance objects. In the image instance segmentation method provided in this application embodiment, due to the correspondence between queries and masquerading instances, only a small number of object instances are activated during instance segmentation, making the large number of queries redundant for each individual image. Furthermore, the fixed number of queries typically does not match the random number of instance objects in the image. In this application embodiment, to further explore the relationship between queries and instance objects, the correspondence between each query and image features is examined from the perspective of query-feature interaction in the cross-attention module. This allows it to be determined that only a few queries in the cross-attention module focus on certain features in the image, while a large number of queries are in an ineffective activation state; that is, these ineffectively activated queries are irrelevant to the object instances. Furthermore, in the masquerading instance segmentation task, a large number of ineffectively activated queries lead to more false positives, increasing task resource consumption.

[0040] It should be noted that the image instance segmentation method provided in this application embodiment can be applied to, but is not limited to, practical application scenarios such as image recognition, instance classification, and change detection of target objects in images. For example, the above method can be applied to the following technical fields: meteorology (e.g., cloud extraction, weather forecasting, weather warning, etc.); natural resources and ecological environment (e.g., weather forecasting, change detection, ecological red line change detection, multi-class change detection, object classification, greenhouse extraction, road network extraction, building extraction, building change detection (satellite, UAV, etc.); water conservancy (e.g., water area change detection, greenhouse extraction, water body extraction (optical, radar), forest area extraction, cage aquaculture extraction, sand mining site extraction, riverside house extraction, dam extraction, photovoltaic power plant extraction, etc.); agriculture Forestry sector (e.g., crop extraction (wheat, rice, potatoes, etc.), drone crop identification (corn, flue-cured tobacco, Job's tears, etc.), plot identification, growth monitoring (index calculation), agricultural yield estimation, pest and disease monitoring, planting suggestion push, etc.); secondary disaster sector (e.g., disaster monitoring, disaster early warning, etc.); life services sector (travel, food delivery, logistics) sector (e.g., travel route planning, travel suggestion push, personnel mobilization, price adjustment, etc.); urban planning sector (e.g., road network extraction (satellite, drone), building extraction, building change detection (satellite, drone), fire protection, etc.).

[0041] According to steps S21 to S22 above, in this embodiment of the application, by acquiring the image to be processed and the query request to be processed, wherein the display content in the image to be processed is the target object, and the query request to be processed includes a location embedding part and a content part, the location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object; further, based on the query request to be processed, the image to be processed is segmented into a disguised instance to obtain the segmentation result corresponding to the target object.

[0042] It is noteworthy that, through the embodiments of this application, by considering the location embedding part and content part involved in the query request during the query-based instance segmentation process, the image to be processed containing the target object is subjected to spoofed instance segmentation to obtain the segmentation result corresponding to the target object. This achieves the purpose of considering the content and location embedding corresponding to the query to perform query-based spoofed instance segmentation of the target object, thereby achieving the technical effect of improving the efficiency and segmentation accuracy of the query-based spoofed instance segmentation process while reducing resource consumption. This solves the technical problem in related technologies that the instance segmentation method with a large number of redundant queries results in low efficiency, low accuracy of segmentation results and high resource consumption.

[0043] In an optional embodiment, in step S22, the image to be processed is segmented into a disguised instance based on the query request to be processed to obtain the segmentation result corresponding to the target object, including the following method steps:

[0044] Step S221: The target camouflage instance segmentation model is used to analyze the query request to be processed and the image to be processed to obtain the segmentation result. The target camouflage instance segmentation model is trained by machine learning using a training dataset, which includes a sample query set and sample images.

[0045] In the optional embodiments described above, the target camouflage instance segmentation model is a neural network model pre-trained using a training dataset via machine learning. The training dataset includes a sample query set and sample images. The training dataset can be pre-collected or updated in real-time during testing of the target camouflage instance segmentation model or during the analysis of the query request and the image to be processed using the target camouflage instance segmentation model. The segmentation result is the camouflage instance segmentation result corresponding to the target object in the image to be processed.

[0046] Specifically, the training process of the target camouflage instance segmentation model and the specific implementation method of using the target camouflage instance segmentation model to analyze the query request to be processed and the image to be processed can be further described in the optional embodiments below.

[0047] In an optional embodiment, the image instance segmentation method further includes the following method steps:

[0048] Step S23: Train the initial camouflaged instance segmentation model using the training dataset to obtain a first prediction result and a second prediction result. The first prediction result is the mask prediction result of the instance in the sample image corresponding to the sample query set, and the second prediction result is the position prediction result of the instance in the sample image corresponding to the sample query set.

[0049] Step S24: Determine the first loss and the second loss using the first prediction result and the second prediction result, wherein the first loss is the similarity measurement loss and the second loss is the cross-entropy loss;

[0050] Step S25: Optimize the model parameters of the initial camouflaged instance segmentation model based on the first loss and the second loss to obtain the target camouflaged instance segmentation model.

[0051] In the above optional embodiments, the initial camouflaged instance segmentation model is a pre-set initial neural network to be trained. The first prediction result is the mask prediction result of the instance corresponding to the sample query set in the sample image obtained by training the initial camouflaged instance segmentation model, and the second prediction result is the position prediction result of the instance corresponding to the sample query set in the sample image obtained by training the initial camouflaged instance segmentation model.

[0052] Furthermore, the similarity metric loss and cross-entropy loss are determined using the first and second prediction results. Based on these losses, the model parameters of the initial camouflage instance segmentation model are optimized to obtain the target camouflage instance segmentation model. Therefore, the target camouflage instance segmentation model considers both similarity metric loss and cross-entropy loss. When segmenting images into camouflage instances, this model can respond to the location embedding portion of the query request to predict the target object's location information in the image, and it can also respond to the content portion of the query request to perform target detection in the image.

[0053] In an optional embodiment, in step S23, the initial masquerading instance segmentation model is trained using the training dataset to obtain a first prediction result, including the following method steps:

[0054] Step S231: Use the initial masquerade instance segmentation model to perform an activity estimation measurement on each sample query in the sample query set to obtain the measurement results;

[0055] Step S232: Determine the number of first sample queries using the measurement results. The number of first sample queries is used to select multiple first sample queries from the sample query set. The multiple first sample queries are multiple sensitive queries.

[0056] Step S233: Perform cross-attention processing on multiple first sample queries based on the number of first sample queries to obtain the first prediction result.

[0057] In the above optional embodiments, the process of training the initial masquerading instance segmentation model using the training dataset can be divided into three stages: First, the initial masquerading instance segmentation model is used to perform an activity estimation measurement on each sample query in the sample query set, that is, to determine the activity of each sample query and obtain a measurement result, wherein the measurement result is used to characterize the activity of each sample query in the sample query set; Second, the measurement result is used to determine the first sample query quantity, and multiple first sample queries are selected from the sample query set according to the first sample query quantity. These multiple first sample queries are multiple sensitive queries (also known as responsive queries, which refer to queries that have corresponding instance objects), and the other sample queries in the sample query set besides the multiple first sample queries are dormant queries (also known as non-responsive queries, which refer to queries that do not have corresponding instance objects); Subsequently, cross-attention processing is performed on the multiple first sample queries based on the first sample query quantity to obtain a first prediction result, wherein the cross-attention processing includes processing of multiple first sample queries by multiple heads.

[0058] In an optional embodiment, in step S231, an activity estimation measurement is performed on each sample query in the sample query set to obtain the measurement result, including the following method steps:

[0059] Step S2311: Obtain the relative entropy and variance of multiple extracted features for each sample query in the sample query set. The relative entropy is used to measure the information content of multiple extracted features, and the variance is used to represent the dispersion of multiple extracted features.

[0060] Step S2312: Perform active estimation measurement based on relative entropy and variance to obtain measurement results.

[0061] In the above optional embodiments, the relative entropy of multiple extracted features for each sample query in the sample query set is used to measure the information content (or energy, where high information content indicates high energy) of the multiple extracted features, and the variance of multiple extracted features for each sample query in the sample query set is used to represent the dispersion (or discreteness) of the multiple extracted features. Further, an activity estimation measurement is performed on each sample query in the sample query set based on the relative entropy and variance to obtain the measurement result.

[0062] It is readily understood that, according to the embodiments of this application, during the process of segmenting camouflaged instances in an image using a target camouflage instance segmentation model, two key indicators (i.e., information content and dispersion) can be used to quantitatively analyze multiple extracted features of each sample query by determining relative entropy and variance. A large relative entropy and variance for a sample query indicates a large information content and dispersion, and thus high activity; conversely, a small relative entropy and variance for a sample query indicates a small information content and dispersion, and thus low activity.

[0063] In an optional embodiment, step S232, determining the first sample query quantity using the measurement results, includes the following method steps:

[0064] Step S2321: Based on the measurement results, determine the number of second sample queries corresponding to each head in the multi-head cross-attention mechanism;

[0065] Step S2322: Select the maximum value from the second sample query counts corresponding to the multiple heads to obtain the first sample query count.

[0066] In the above optional embodiments, based on the measurement results of the activity estimation measurement of each sample query, the second sample query number corresponding to each head (also called a magnetic head) in the multi-head cross-attention mechanism is determined, wherein the second sample query number is the number of candidate sample queries corresponding to each head initially determined according to the measurement results. The maximum value is selected from the second sample query numbers corresponding to the multiple heads respectively to obtain the first sample query number, which is the number of target sample queries (i.e., multiple sensitive queries) to be processed corresponding to the multi-head attention processing.

[0067] In an optional embodiment, in step S233, multi-head cross-attention processing is performed on multiple first sample queries based on the number of first sample queries to obtain a first prediction result, including the following method steps:

[0068] Step S2331: Select multiple first sample queries from the sample query set based on the first sample query count, and discard multiple second sample queries, wherein the multiple second sample queries are multiple dormant queries;

[0069] Step S2332: Perform multi-head cross-attention processing on multiple first sample queries to obtain the first prediction result.

[0070] In the above optional embodiments, the multiple first sample queries are multiple sensitive queries, and the multiple second sample queries are multiple dormant queries. The multiple sensitive queries are considered valid queries in the sample query set, and the multiple dormant queries are considered invalid queries (i.e., redundant queries) in the sample query set. Based on the number of the first sample queries, multiple valid queries are selected from the sample query set, and multiple invalid queries are discarded. Further, multi-head cross-attention processing is performed on the first sample queries to obtain the mask prediction results of the instances in the sample image corresponding to the sample query set.

[0071] In an optional embodiment, in step S23, the initial masquerading instance segmentation model is trained using the training dataset to obtain a second prediction result, including the following method steps:

[0072] Step S234: The initial camouflaged instance segmentation model is used to analyze the location embedding part of each sample query in the sample query set to obtain the second prediction result. The location embedding part of each sample query is extracted in advance from the boundary points of the target instance in the sample image.

[0073] In the above optional embodiments, the location embedding part of each sample query in the sample query set is pre-extracted from the boundary points of the target instance in the sample image. During the training of the initial camouflaged instance segmentation model using the training dataset, the location embedding part of each sample query in the sample query set is analyzed using the initial camouflaged instance segmentation model to obtain the location prediction result of the instance in the sample image corresponding to the sample query set.

[0074] It is noteworthy that by considering the above-mentioned location embedding part in the embodiments of this application, the interpretability of the location embedding corresponding to the sample query can be improved, thereby obtaining more effective location information and performing instance query matching more accurately.

[0075] As described in the above optional embodiments, it is understood that the image instance segmentation method provided according to the embodiments of this application can effectively collect image semantics from the entire training dataset and query all instance objects, while deleting redundant queries in specific scenarios to eliminate false alarm retrieval.

[0076] Furthermore, the method provided in this application embodiment also uses entropy and variance to determine energy and dispersion to quantify multiple extracted features of each sample query, thereby determining the query category of each sample query, that is, whether each sample query belongs to a sensitive query or a dormant query, so as to discard multiple dormant queries (i.e. invalid queries or redundant queries) in the sample query set during the training process.

[0077] It is easy to understand that, compared with the fixed selection strategy in the prior art, the image instance segmentation method provided in this application embodiment can adopt different dynamic sampling strategies for multiple heads according to the entropy and variance of each query. Therefore, the above-mentioned method provided in this application embodiment can be applied to various scene images containing different numbers of instance objects.

[0078] Furthermore, location information is particularly important in masquerading instance segmentation tasks. Accurate location information can lead to more discriminative object representations, especially making occluded instance objects more clearly represented. However, in commonly used instance segmentation models in the prior art (such as OSFormer), the location encoding of object queries is randomly initialized, which makes it difficult to accurately predict the instance location during the query process. To solve the above problem, the image instance segmentation method provided in this application uses the boundary position of the instance object in the query input as a condition to determine the dynamic location encoding, thereby enabling explicit prediction of the query implicit in the instance boundary points, and thus performing more accurate matching between the query and the instance.

[0079] Figure 3 This is a schematic diagram of an optional adaptive query architecture according to an embodiment of this application, such as... Figure 3As shown, the process of training the initial masquerading instance segmentation model using the training dataset to obtain the first prediction result is divided into three stages. The first stage is the dynamic evaluation process of the query, the second stage is the multi-head adaptive query selection process, and the third stage is the interaction process between sensitive queries and features.

[0080] like Figure 3 As shown, X represents the feature space, where features are extracted from the backbone network or enhanced by the corresponding model encoder (such as the Transformer model encoder). Q represents a set of sample queries, also known as the sample query set, which contains m sample queries ( Figure 3 (The example shown is a query set of 8 samples). The feature set extracted from N training samples (i.e., the multiple extracted features mentioned above) is denoted as XN. The feature set XN is encoded to obtain the sample query set Q mentioned above.

[0081] like Figure 3 As shown, the location embedding of the query is denoted as Qp, and the content is denoted as Qc. The feature space is mapped to the d-dimensional representation of the third stage through a linear layer, with mapping keys K and V respectively. The query location embedding Qp and query content Qc correspond to the location embedding Kp and content Kc of the feature space X in K, respectively. Based on the above, the sample query set Q is obtained by the following formula (1):

[0082] Q=Qc+Qp=q1, q2,..., qL Formula (1)

[0083] During the query process, the attention of the i-th query sample is represented as a kernel-smoothed probability form, as shown in the following formula (2):

[0084]

[0085] In the above formula (2), qi, ki and vi represent the i-th row of Q, K and V respectively, and P(kj, qi) represents the attention probability of the i-th query sample to all keys. Specifically, P(kj, qi) is calculated according to the following formula (3):

[0086]

[0087] It is readily apparent that for a complete query mapping, the number of instance objects in the image is typically much smaller than the number of query samples. In other words, most queries in existing technologies are redundant, leading to false positives in all complete mapping methods due to redundant queries. This problem of false positives caused by redundant queries is even more pronounced in camouflaged object segmentation tasks, where foreground object instances and background objects share a high degree of similarity.

[0088] like Figure 3As shown in the first stage, the vertical axis of the image is P(kj, qi), query A represents a sensitive query, and query B represents a dormant query. It is easy to see that not every query retrieved from a sample is valuable; a large number of queries do not focus on any region during the interaction. Furthermore, the K and Q of cross-attention lack prior knowledge of location embedding relevance, meaning that current techniques do not consider the interpretability of location embeddings.

[0089] Still as Figure 3 As shown, in the first stage, according to the calculation method of P(kj, qi), active queries push the attention probability distribution of the response query samples, represented by clicks, away from the uniform distribution. That is, the query process of the instance object pays special attention to certain elements in key K, exhibiting high activity in responses to these elements. To quantitatively analyze the activity of sample queries, two key indicators are introduced: information content and dispersion. Information content is measured by relative entropy; a smaller relative entropy indicates a lower similarity between the P(kj, qi) corresponding to the query sample and the uniform distribution, while a larger relative entropy indicates a higher similarity. Dispersion is measured by variance; features extracted by sensitive queries have greater variance, while features extracted by dormant queries have smaller variance.

[0090] Existing Top-m value generation strategies have a key drawback: the generated m value (i.e., the number of queries for the first sample) is fixed. However, the instances contained in an image vary greatly, and a fixed m value is meaningless for individual image instances. To address this, embodiments of this application employ an adaptive Top-m query selection strategy, dynamically selecting the corresponding number of queries for each sample query.

[0091] In such Figure 3 In the second stage shown, the most typical query sample corresponding to each head is first calculated to obtain the sensitive query of the image. The m value corresponding to the i-th head is calculated as shown in the following formula (4):

[0092]

[0093] To perform mini-batch training on N samples, the final value of n is calculated according to the following formula (5):

[0094] m = max{mi, ..., m(n-1)} i＝1，2，...，N Formula (5)

[0095] In other words, the largest m value in the mini-batch samples is selected as the final m value.

[0096] In such Figure 3The third stage, as shown, collects as much instance information as possible during the query process while ignoring as many dormant queries as possible. In other words, it performs cross-attention on sensitive queries, while dormant queries do not interact with features to avoid the impact of redundant queries. Specifically, the strategy of the above process in this application is as follows: when querying samples... q When i belongs to Top-m, the attention representation A(qi, K, V) of the query sample is calculated using the method shown in formula (2); when the query sample qi belongs to Top-m, the attention representation A(qi, K, V) of the query sample is 0.

[0097] It is easy to understand that in the third stage mentioned above, the attention representation A(qi, K, V) for dormant queries is set to 0, which can discard dormant queries when performing multi-head cross-attention processing on multiple sensitive queries, and further avoid the impact of redundant queries.

[0098] Furthermore, in the process of training the initial camouflaged instance segmentation model using the training dataset to obtain the second prediction result, boundary position embedding information is introduced for the sample query. The position embedding of each sample query is derived from the important boundary points of the object instance in the sample image set H, as shown in the following formula (6):

[0099] Qp = Top - s(H) Formula (6)

[0100] In the above formula (6), Top-s equals the number of queries. Since H retains more boundary features of object instances, the feature space X transforms the boundary features through boundary extraction. In the embodiment of this application, referring to the image enhancement method, considering the isotropic properties of the image, the second-order Laplacian operator is used to compensate for the image contour and enhance the transition part of the boundary and features. The above process is shown in formula (7):

[0101]

[0102] In formula (7) above, ks is the kernel size, which is taken as 3 in this example. Due to the use of a second-order Laplace operator, the above process is more sensitive to noise. Therefore, a fuzzy filter f is set in the above process. med (·) Perform noise reduction.

[0103] Furthermore, considering the continuity of the boundary, directly selecting the top-s features would trigger a clustering effect, causing the top-s features to fail. Therefore, this application filters out the highest local response points in the patches. It is easy to understand that boundary information at different scales can be aggregated to enhance the transition between the boundary and the features.

[0104] As an extension of the embodiments of this application, in the image instance segmentation method, a structure consisting of a backbone and a Transformer encoder is used for feature extraction. The backbone can be a convolutional neural network, and the Transformer encoder is used to enhance the features captured by the backbone.

[0105] It is readily understood that the method provided in this application utilizes an adaptive query selection mechanism to improve the query-based masquerading instance segmentation model by suppressing invalid queries. First, considering the masquerading instance segmentation model is prone to false positives (i.e., all queries are valid for all instances in the entire dataset), this leads to high redundancy in the current query scheme for instance segmentation in specific scenarios. Therefore, this application's embodiments enable the query process to effectively collect semantics from the entire dataset and retrieve all instances, while simultaneously removing redundant queries in specific scenarios to eliminate false positives. Second, this application's embodiments provide an interpretable query location embedding mechanism with masquerading instance boundaries, thereby improving the interpretability of query location embedding, leading to more effective location information and more accurate instance query matching.

[0106] In summary, the image instance segmentation method proposed in this application adopts an adaptive query selection mechanism, which improves the query-based masquerading instance segmentation model by continuously performing invalid queries, and considers two basic relationships between queries and masquerading instances: query redundancy and position embedding interpretability.

[0107] In one alternative embodiment, a graphical user interface is provided via a cloud-native platform. The content displayed by the graphical user interface at least partially includes an object masquerading instance segmentation scene. The image instance segmentation method further includes the following method steps:

[0108] Step S261: In response to a first touch operation applied to the graphical user interface, select an image to be processed from multiple candidate images;

[0109] Step S262: In response to a second touch operation applied to the graphical user interface, select a target masquerading instance segmentation model from multiple candidate instance segmentation models;

[0110] Step S263: In response to the third touch operation applied to the graphical user interface, the target camouflage instance segmentation model is used to analyze the query request to be processed and the image to be processed to obtain the segmentation result;

[0111] Step S264: Display the segmentation results within the graphical user interface.

[0112] In the above optional embodiments, the object camouflage instance segmentation scenario displayed by the above graphical user interface can be a camouflage instance segmentation scenario in tasks such as image recognition, instance classification, and change detection of target objects in an image.

[0113] The graphical user interface described above also includes a first control (or a first touch area). When a first touch operation is detected on the first control (or the first touch area), an image to be processed is selected from multiple candidate images. For example, the optional embodiments provided in this application can be applied to image recognition application scenarios. In this application scenario, the multiple candidate images are multiple candidate images captured by an image acquisition device during the camouflage instance segmentation process in image recognition. All of these multiple candidate images display camouflage instance objects. Correspondingly, the image to be processed is the image to be processed. The image to be processed displays a target camouflage instance object.

[0114] The graphical user interface also includes a second control (or a second touch area). When a second touch operation is detected on the second control (or the second touch area), a target camouflage instance segmentation model is selected from multiple candidate instance segmentation models. These multiple candidate instance segmentation models are pre-trained neural network models. By performing the second touch operation, the user can specify the target camouflage instance segmentation model to be used from these multiple candidate instance segmentation models.

[0115] The graphical user interface also includes a third control (or a third touch area). When a third touch operation is detected on the third control (or the third touch area), a target camouflage instance segmentation model is used to analyze the query request to be processed and the image to be processed to obtain a segmentation result. The segmentation result is the analysis result of the target camouflage instance segmentation model corresponding to the camouflage instance object in the image to be processed. The query request to be processed can be a query request entered by the user or a query request automatically determined based on the image to be processed.

[0116] Furthermore, the segmentation results are displayed to the user within the graphical user interface. Through steps S261 to S264, the user can specify the image to be processed, specify the target masquerading instance segmentation model to be used, and trigger the target masquerading instance segmentation model to automatically analyze the query request and the image to be processed by the graphical user interface that displays the masquerading instance segmentation scene. Thus, the user can automatically obtain the corresponding segmentation results through the touch operation of the graphical user interface.

[0117] It should be noted that the first, second, and third touch operations described above can all be operations performed by a user touching the display screen of the terminal device with their finger and interacting with the terminal device. These touch operations can include single-point touch and multi-point touch, where each touch point can be interacted with by clicking, long-pressing, pressing hard, or swiping. The first, second, and third touch operations can also be implemented using input devices such as a mouse or keyboard.

[0118] Under the aforementioned operating environment, this application provides the following: Figure 4 This illustrates an image instance segmentation method. Figure 4 This is a flowchart of another image instance segmentation method according to an embodiment of this application, such as... Figure 4 As shown, the image instance segmentation method includes:

[0119] Step S41: Obtain the image of the urban building complex to be processed and the query request to be processed. The display content in the image of the urban building complex to be processed is the target building. The query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target building, and the content part is used to perform target detection on the target building.

[0120] Step S42: Based on the query request to be processed, perform camouflage instance segmentation on the image of the city building complex to be processed to obtain the segmentation result corresponding to the target building.

[0121] In the above embodiments of this application, the target building in the urban building complex image to be processed is the segmentation target for camouflage instance segmentation. The query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target building, and the content part is used to detect the target building. That is, according to the method provided by the embodiments of this application, in the query-based camouflage instance segmentation process, a query paradigm that considers both location embedding and query content is adopted to perform camouflage instance segmentation on the urban building complex image to be processed, thereby obtaining the segmentation result corresponding to the target building.

[0122] According to steps S41 to S42 above, in this embodiment of the application, by acquiring an image of a city building complex to be processed and a query request to be processed, wherein the display content in the image of the city building complex to be processed is the target building, and the query request to be processed includes a location embedding part and a content part, the location embedding part is used to predict the location information of the target building, and the content part is used to perform target detection on the target building; further, based on the query request to be processed, the image of the city building complex to be processed is segmented into camouflaged instances to obtain the segmentation result corresponding to the target building.

[0123] It is noteworthy that, through the embodiments of this application, by considering the location embedding part and content part involved in the query request during the query-based instance segmentation process, the image of the urban building complex containing the target building is subjected to dummy instance segmentation, and the segmentation result corresponding to the target building is obtained. This achieves the purpose of considering the content and location embedding corresponding to the query to perform query-based dummy instance segmentation of the target building, thereby achieving the technical effect of improving the efficiency and segmentation accuracy of the query-based dummy instance segmentation process while reducing resource consumption. This solves the technical problem in related technologies that the instance segmentation method with a large number of redundant queries results in low efficiency, low accuracy of segmentation results and high resource consumption.

[0124] In one alternative embodiment, a graphical user interface is provided via a cloud-native platform. The content displayed by the graphical user interface at least partially includes a scene of camouflaged instance segmentation of city buildings. The image instance segmentation method further includes the following method steps:

[0125] Step S431: In response to the first touch operation applied to the graphical user interface, select the city building cluster image to be processed from multiple candidate city building cluster images;

[0126] Step S432: In response to a second touch operation applied to the graphical user interface, select a target masquerading instance segmentation model from multiple candidate instance segmentation models;

[0127] Step S433: In response to the third touch operation applied to the graphical user interface, the target camouflage instance segmentation model is used to analyze the query request to be processed and the image of the city building complex to be processed to obtain the segmentation result;

[0128] Step S434: Display the segmentation results within the graphical user interface.

[0129] In the above optional embodiments, the camouflaged instance segmentation scene of urban buildings displayed in the above graphical user interface can be a camouflaged instance segmentation scene in tasks such as building group image recognition, building group classification, and building group change detection.

[0130] The graphical user interface also includes a first control (or a first touch area). When a first touch operation is detected on the first control (or the first touch area), a city building cluster image to be processed is selected from multiple candidate city building cluster images. The multiple candidate city building cluster images are multiple images captured by an image acquisition device in the camouflage instance segmentation scene. Camouflage instance objects are displayed in all of these multiple candidate city building cluster images. The city building cluster image to be processed displays the target camouflage instance object.

[0131] The graphical user interface also includes a second control (or a second touch area). When a second touch operation is detected on the second control (or the second touch area), a target camouflage instance segmentation model is selected from multiple candidate instance segmentation models. These multiple candidate instance segmentation models are pre-trained neural network models. By performing the second touch operation, the user can specify the target camouflage instance segmentation model to be used from these multiple candidate instance segmentation models.

[0132] The aforementioned graphical user interface also includes a third control (or a third touch area). When a third touch operation is detected on the third control (or the third touch area), a target camouflage instance segmentation model is used to analyze the query request to be processed and the image of the urban building complex to be processed, obtaining a segmentation result. The segmentation result is the analysis result of the target camouflage instance segmentation model corresponding to the camouflage instance object in the image of the urban building complex to be processed. The aforementioned query request to be processed can be a query request input by the user or a query request automatically determined based on the image of the urban building complex to be processed.

[0133] Furthermore, the segmentation results are displayed to the user within the graphical user interface. Through steps S261 to S264, the user can specify the image to be processed, specify the target camouflage instance segmentation model to be used, and trigger the target camouflage instance segmentation model to automatically analyze the query request to be processed and the image of the urban building complex to be processed by touching the graphical user interface. Thus, the user can automatically obtain the corresponding segmentation results through the touch operation of the graphical user interface.

[0134] It should be noted that the first, second, and third touch operations described above can all be operations performed by a user touching the display screen of the terminal device with their finger and interacting with the terminal device. These touch operations can include single-point touch and multi-point touch, where each touch point can be interacted with by clicking, long-pressing, pressing hard, or swiping. The first, second, and third touch operations can also be implemented using input devices such as a mouse or keyboard.

[0135] The image instance segmentation method provided in this application is also applicable to ship instance segmentation scenarios in maritime images. The image instance segmentation method in this scenario includes:

[0136] Step S451: Obtain the image of the floating object to be processed and the query request to be processed. The content displayed in the image of the floating object to be processed is the target vessel. The query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target vessel, and the content part is used to detect the target vessel.

[0137] Step S452: Based on the query request to be processed, perform camouflage instance segmentation on the image of floating objects at sea to be processed, and obtain the segmentation result corresponding to the target vessel.

[0138] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0140] Example 2

[0141] According to an embodiment of this application, an apparatus embodiment for implementing the above-described image instance segmentation method is also provided. Figure 5 This is a schematic diagram of the structure of an image instance segmentation device according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0142] The acquisition module 501 is used to acquire the image to be processed and the query request to be processed. The display content in the image to be processed is the target object. The query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object.

[0143] The segmentation module 502 is used to perform masquerade instance segmentation on the image to be processed based on the query request to be processed, and obtain the segmentation result corresponding to the target object.

[0144] Optionally, the segmentation module 502 is further configured to: analyze the query request to be processed and the image to be processed using a target camouflage instance segmentation model to obtain a segmentation result, wherein the target camouflage instance segmentation model is trained by machine learning using a training dataset, and the training dataset includes: a sample query set and sample images.

[0145] Optionally, Figure 6 This is a schematic diagram of an optional image instance segmentation device according to an embodiment of this application, such as... Figure 6 As shown, the device includes Figure 5 In addition to all the modules shown, the system also includes: a training module 503, used to train the initial camouflaged instance segmentation model using a training dataset to obtain a first prediction result and a second prediction result, wherein the first prediction result is the mask prediction result of the instance in the sample image corresponding to the sample query set, and the second prediction result is the position prediction result of the instance in the sample image corresponding to the sample query set; a determination module 504, used to determine a first loss and a second loss using the first prediction result and the second prediction result, wherein the first loss is a similarity metric loss, and the second loss is a cross-entropy loss; and an optimization module 505, used to optimize the model parameters of the initial camouflaged instance segmentation model based on the first loss and the second loss to obtain a target camouflaged instance segmentation model.

[0146] Optionally, the training module 503 is further configured to: use an initial masquerading instance segmentation model to perform an activity estimation measurement on each sample query in the sample query set to obtain a measurement result; use the measurement result to determine the number of first sample queries, wherein the number of first sample queries is used to select multiple first sample queries from the sample query set, and the multiple first sample queries are multiple sensitive queries; and perform cross-attention processing on the multiple first sample queries based on the number of first sample queries to obtain a first prediction result.

[0147] Optionally, the training module 503 is further configured to: obtain the relative entropy and variance of multiple extracted features for each sample query in the sample query set, wherein the relative entropy is used to measure the information content of multiple extracted features, and the variance is used to represent the dispersion of multiple extracted features; and perform an activity estimation measurement based on the relative entropy and variance to obtain the measurement result.

[0148] Optionally, the training module 503 is further configured to: determine the number of second sample queries corresponding to each head in the multi-head cross-attention mechanism based on the measurement results; and select the maximum value from the number of second sample queries corresponding to multiple heads to obtain the number of first sample queries.

[0149] Optionally, the training module 503 is further configured to: select multiple first sample queries from the sample query set based on the number of first sample queries, and discard multiple second sample queries, wherein the multiple second sample queries are multiple dormant queries; perform multi-head cross-attention processing on the multiple first sample queries to obtain a first prediction result.

[0150] Optionally, the training module 503 is further configured to: analyze the location embedding part of each sample query in the sample query set using the initial camouflaged instance segmentation model to obtain a second prediction result, wherein the location embedding part of each sample query is pre-extracted from the boundary points of the target instance in the sample image.

[0151] Optionally, Figure 7 This is a schematic diagram of another optional image instance segmentation device according to an embodiment of this application, such as... Figure 7 As shown, the device includes Figure 6 In addition to all the modules shown, it also includes: a display module 506, which is used to select an image to be processed from multiple candidate images in response to a first touch operation applied to the graphical user interface; select a target camouflage instance segmentation model from multiple candidate instance segmentation models in response to a second touch operation applied to the graphical user interface; analyze the query request to be processed and the image to be processed using the target camouflage instance segmentation model in response to a third touch operation applied to the graphical user interface to obtain a segmentation result; and display the segmentation result within the graphical user interface.

[0152] It should be noted that the acquisition module and the segmentation module mentioned above correspond to steps S21 to S22 in Embodiment 1. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0153] In this embodiment, an image to be processed and a query request to be processed are obtained. The image to be processed contains the target object, and the query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object. Furthermore, based on the query request to be processed, the image to be processed is segmented into a disguised instance to obtain the segmentation result corresponding to the target object.

[0154] It is noteworthy that, through the embodiments of this application, by considering the location embedding part and content part involved in the query request during the query-based instance segmentation process, the image to be processed containing the target object is subjected to spoofed instance segmentation to obtain the segmentation result corresponding to the target object. This achieves the purpose of considering the content and location embedding corresponding to the query to perform query-based spoofed instance segmentation of the target object, thereby achieving the technical effect of improving the efficiency and segmentation accuracy of the query-based spoofed instance segmentation process while reducing resource consumption. This solves the technical problem in related technologies that the instance segmentation method with a large number of redundant queries results in low efficiency, low accuracy of segmentation results and high resource consumption.

[0155] According to an embodiment of this application, an apparatus embodiment for implementing another image instance segmentation method described above is also provided. Figure 8 This is a schematic diagram of another image instance segmentation device according to an embodiment of this application, such as... Figure 8 As shown, the device includes:

[0156] The acquisition module 801 is used to acquire an image of a city building complex to be processed and a query request to be processed. The display content in the image of the city building complex to be processed is the target building. The query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target building, and the content part is used to perform target detection on the target building.

[0157] The segmentation module 802 is used to perform camouflage instance segmentation on the image of the urban building complex to be processed based on the query request to be processed, and to obtain the segmentation result corresponding to the target building.

[0158] Optionally, Figure 9 This is a schematic diagram of another optional image instance segmentation device according to an embodiment of this application, such as... Figure 9 As shown, the device includes Figure 8 In addition to all the modules shown, it also includes: a display module 803, which is used to select a city building cluster image to be processed from multiple candidate city building cluster images in response to a first touch operation applied to the graphical user interface; select a target camouflage instance segmentation model from multiple candidate instance segmentation models in response to a second touch operation applied to the graphical user interface; analyze the query request to be processed and the city building cluster image to be processed using the target camouflage instance segmentation model in response to a third touch operation applied to the graphical user interface to obtain a segmentation result; and display the segmentation result within the graphical user interface.

[0159] It should be noted that the acquisition module and the segmentation module mentioned above correspond to steps S41 to S42 in Embodiment 1. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0160] In this embodiment, an image of a city building complex to be processed and a query request to be processed are obtained. The display content in the image of the city building complex to be processed is the target building. The query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target building, and the content part is used to detect the target building. Furthermore, based on the query request to be processed, the image of the city building complex to be processed is segmented into camouflaged instances to obtain the segmentation result corresponding to the target building.

[0161] It is noteworthy that, through the embodiments of this application, by considering the location embedding part and content part involved in the query request during the query-based instance segmentation process, the image of the urban building complex containing the target building is subjected to dummy instance segmentation, and the segmentation result corresponding to the target building is obtained. This achieves the purpose of considering the content and location embedding corresponding to the query to perform query-based dummy instance segmentation of the target building, thereby achieving the technical effect of improving the efficiency and segmentation accuracy of the query-based dummy instance segmentation process while reducing resource consumption. This solves the technical problem in related technologies that the instance segmentation method with a large number of redundant queries results in low efficiency, low accuracy of segmentation results and high resource consumption.

[0162] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0163] Example 3

[0164] According to an embodiment of this application, an embodiment of an electronic device is also provided. This electronic device can be any computing device in a group of computing devices. The electronic device includes: a processor and a memory, wherein:

[0165] The memory, connected to the processor, is used to provide the processor with instructions to perform the following processing steps: acquiring an image to be processed and a query request to be processed, wherein the display content in the image to be processed is a target object, and the query request to be processed includes a location embedding part and a content part, wherein the location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object; and performing camouflage instance segmentation on the image to be processed based on the query request to be processed to obtain the segmentation result corresponding to the target object.

[0166] In this embodiment, an image to be processed and a query request to be processed are obtained. The image to be processed contains the target object, and the query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object. Furthermore, based on the query request to be processed, the image to be processed is segmented into a disguised instance to obtain the segmentation result corresponding to the target object.

[0167] It is noteworthy that, through the embodiments of this application, by considering the location embedding part and content part involved in the query request during the query-based instance segmentation process, the image to be processed containing the target object is subjected to spoofed instance segmentation to obtain the segmentation result corresponding to the target object. This achieves the purpose of considering the content and location embedding corresponding to the query to perform query-based spoofed instance segmentation of the target object, thereby achieving the technical effect of improving the efficiency and segmentation accuracy of the query-based spoofed instance segmentation process while reducing resource consumption. This solves the technical problem in related technologies that the instance segmentation method with a large number of redundant queries results in low efficiency, low accuracy of segmentation results and high resource consumption.

[0168] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0169] Example 4

[0170] Embodiments of this application may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced by a mobile terminal or other terminal device.

[0171] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0172] In this embodiment, the computer terminal described above can execute the program code for the following steps in the image instance segmentation method: obtaining an image to be processed and a query request to be processed, wherein the display content in the image to be processed is a target object, and the query request to be processed includes a location embedding part and a content part, wherein the location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object; and performing camouflage instance segmentation on the image to be processed based on the query request to be processed to obtain the segmentation result corresponding to the target object.

[0173] Optionally, Figure 10 This is a structural block diagram of another computer terminal according to an embodiment of this application, such as... Figure 10As shown, the computer terminal may include one or more (only one is shown in the figure) processors 122, memory 124, and peripheral interfaces 126.

[0174] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image instance segmentation method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image instance segmentation method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0175] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: acquiring an image to be processed and a query request to be processed, wherein the display content in the image to be processed is the target object, and the query request to be processed includes a location embedding part and a content part, wherein the location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object; and performing camouflage instance segmentation on the image to be processed based on the query request to be processed to obtain the segmentation result corresponding to the target object.

[0176] Optionally, the processor may also execute program code that performs the following steps: using a target camouflage instance segmentation model to analyze the query request to be processed and the image to be processed to obtain a segmentation result, wherein the target camouflage instance segmentation model is trained by machine learning using a training dataset, and the training dataset includes: a sample query set and sample images.

[0177] Optionally, the processor may also execute program code for the following steps: training the initial camouflaged instance segmentation model using a training dataset to obtain a first prediction result and a second prediction result, wherein the first prediction result is the mask prediction result of the instance in the sample image corresponding to the sample query set, and the second prediction result is the position prediction result of the instance in the sample image corresponding to the sample query set; determining a first loss and a second loss using the first prediction result and the second prediction result, wherein the first loss is a similarity metric loss, and the second loss is a cross-entropy loss; optimizing the model parameters of the initial camouflaged instance segmentation model based on the first loss and the second loss to obtain the target camouflaged instance segmentation model.

[0178] Optionally, the processor may also execute program code that performs the following steps: using an initial masquerade instance segmentation model to perform an activity estimation measurement on each sample query in the sample query set to obtain a measurement result; using the measurement result to determine the number of first sample queries, wherein the number of first sample queries is used to select multiple first sample queries from the sample query set, and the multiple first sample queries are multiple sensitive queries; and performing cross-attention processing on the multiple first sample queries based on the number of first sample queries to obtain a first prediction result.

[0179] Optionally, the processor may also execute program code that performs the following steps: obtaining the relative entropy and variance of multiple extracted features for each sample query in the sample query set, wherein the relative entropy is used to measure the information content of the multiple extracted features, and the variance is used to represent the dispersion of the multiple extracted features; performing an activity estimation measurement based on the relative entropy and variance to obtain the measurement result.

[0180] Optionally, the processor may also execute program code that performs the following steps: based on the measurement results, determine the number of second sample queries corresponding to each head in the multi-head cross-attention mechanism; select the maximum value from the number of second sample queries corresponding to multiple heads respectively to obtain the number of first sample queries.

[0181] Optionally, the processor may also execute program code that performs the following steps: selects multiple first sample queries from the sample query set based on the first sample query count, and discards multiple second sample queries, wherein the multiple second sample queries are multiple dormant queries; performs multi-head cross-attention processing on the multiple first sample queries to obtain a first prediction result.

[0182] Optionally, the processor may also execute program code that performs the following steps: analyzes the location embedding of each sample query in the sample query set using an initial masquerading instance segmentation model to obtain a second prediction result, wherein the location embedding of each sample query is pre-extracted from the boundary points of the target instance in the sample image.

[0183] Optionally, the processor may also execute program code that performs the following steps: in response to a first touch operation applied to the graphical user interface, selects an image to be processed from multiple candidate images; in response to a second touch operation applied to the graphical user interface, selects a target camouflage instance segmentation model from multiple candidate instance segmentation models; in response to a third touch operation applied to the graphical user interface, analyzes the query request to be processed and the image to be processed using the target camouflage instance segmentation model to obtain a segmentation result; and displays the segmentation result within the graphical user interface.

[0184] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: acquiring an image of a city building complex to be processed and a query request to be processed, wherein the display content in the image of the city building complex to be processed is the target building, and the query request to be processed includes a location embedding part and a content part, wherein the location embedding part is used to predict the location information of the target building, and the content part is used to perform target detection on the target building; and performing camouflage instance segmentation on the image of the city building complex to be processed based on the query request to be processed to obtain the segmentation result corresponding to the target building.

[0185] Optionally, the processor may also execute program code that performs the following steps: in response to a first touch operation applied to the graphical user interface, selects an image of a city building cluster to be processed from multiple candidate images of city building clusters; in response to a second touch operation applied to the graphical user interface, selects a target camouflage instance segmentation model from multiple candidate instance segmentation models; in response to a third touch operation applied to the graphical user interface, analyzes the query request to be processed and the image of the city building cluster to be processed using the target camouflage instance segmentation model to obtain a segmentation result; and displays the segmentation result within the graphical user interface.

[0186] In this embodiment, an image to be processed and a query request to be processed are obtained. The image to be processed contains the target object, and the query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object. Furthermore, based on the query request to be processed, the image to be processed is segmented into a disguised instance to obtain the segmentation result corresponding to the target object.

[0187] It is noteworthy that, through the embodiments of this application, by considering the location embedding part and content part involved in the query request during the query-based instance segmentation process, the image to be processed containing the target object is subjected to spoofed instance segmentation to obtain the segmentation result corresponding to the target object. This achieves the purpose of considering the content and location embedding corresponding to the query to perform query-based spoofed instance segmentation of the target object, thereby achieving the technical effect of improving the efficiency and segmentation accuracy of the query-based spoofed instance segmentation process while reducing resource consumption. This solves the technical problem in related technologies that the instance segmentation method with a large number of redundant queries results in low efficiency, low accuracy of segmentation results and high resource consumption.

[0188] Those skilled in the art will understand that Figure 10 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Figure 10This does not limit the structure of the aforementioned electronic devices. For example, a computer terminal may also include components that are more... Figure 10 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 10 The different configurations shown.

[0189] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0190] According to an embodiment of this application, an embodiment of a computer-readable storage medium is also provided. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the image instance segmentation method provided in Embodiment 1.

[0191] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0192] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring an image to be processed and a query request to be processed, wherein the display content in the image to be processed is a target object, and the query request to be processed includes a location embedding part and a content part, wherein the location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object; and performing camouflage instance segmentation on the image to be processed based on the query request to be processed to obtain the segmentation result corresponding to the target object.

[0193] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: analyzing the query request to be processed and the image to be processed using a target camouflage instance segmentation model to obtain a segmentation result, wherein the target camouflage instance segmentation model is trained by machine learning using a training dataset, and the training dataset includes: a sample query set and sample images.

[0194] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: training an initial camouflaged instance segmentation model using a training dataset to obtain a first prediction result and a second prediction result, wherein the first prediction result is a mask prediction result of the instance in the sample image corresponding to the sample query set, and the second prediction result is a position prediction result of the instance in the sample image corresponding to the sample query set; determining a first loss and a second loss using the first prediction result and the second prediction result, wherein the first loss is a similarity metric loss, and the second loss is a cross-entropy loss; optimizing the model parameters of the initial camouflaged instance segmentation model based on the first loss and the second loss to obtain a target camouflaged instance segmentation model.

[0195] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: using an initial masquerading instance segmentation model to perform an activity estimation measurement on each sample query in the sample query set to obtain a measurement result; using the measurement result to determine a first sample query quantity, wherein the first sample query quantity is used to select multiple first sample queries from the sample query set, and the multiple first sample queries are multiple sensitive queries; and performing cross-attention processing on the multiple first sample queries based on the first sample query quantity to obtain a first prediction result.

[0196] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining the relative entropy and variance of multiple extracted features for each sample query in the sample query set, wherein the relative entropy is used to measure the information content of the multiple extracted features, and the variance is used to represent the dispersion of the multiple extracted features; performing an activity estimation measurement based on the relative entropy and variance to obtain the measurement result.

[0197] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining the number of second sample queries corresponding to each head in the multi-head cross-attention mechanism based on the measurement results; selecting the maximum value from the number of second sample queries corresponding to multiple heads respectively to obtain the number of first sample queries.

[0198] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: selecting multiple first sample queries from the sample query set based on the first sample query quantity, and discarding multiple second sample queries, wherein the multiple second sample queries are multiple dormant queries; performing multi-head cross-attention processing on the multiple first sample queries to obtain a first prediction result.

[0199] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: analyzing the location embedding part of each sample query in the sample query set using an initial masquerading instance segmentation model to obtain a second prediction result, wherein the location embedding part of each sample query is pre-extracted from the boundary points of the target instance in the sample image.

[0200] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: in response to a first touch operation applied to the graphical user interface, selecting an image to be processed from multiple candidate images; in response to a second touch operation applied to the graphical user interface, selecting a target camouflage instance segmentation model from multiple candidate instance segmentation models; in response to a third touch operation applied to the graphical user interface, analyzing the query request to be processed and the image to be processed using the target camouflage instance segmentation model to obtain a segmentation result; and displaying the segmentation result within the graphical user interface.

[0201] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring an image of a city building complex to be processed and a query request to be processed, wherein the display content in the image of the city building complex to be processed is: a target building, and the query request to be processed includes: a location embedding part and a content part, wherein the location embedding part is used to predict the location information of the target building, and the content part is used to perform target detection on the target building; performing camouflage instance segmentation on the image of the city building complex to be processed based on the query request to be processed, and obtaining the segmentation result corresponding to the target building.

[0202] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: in response to a first touch operation applied to the graphical user interface, selecting a city building cluster image to be processed from multiple candidate city building cluster images; in response to a second touch operation applied to the graphical user interface, selecting a target camouflage instance segmentation model from multiple candidate instance segmentation models; in response to a third touch operation applied to the graphical user interface, analyzing the query request to be processed and the city building cluster image to be processed using the target camouflage instance segmentation model to obtain a segmentation result; and displaying the segmentation result within the graphical user interface.

[0203] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0204] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0206] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0207] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0208] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, ROM, RAM, portable hard drives, magnetic disks, or optical disks.

[0209] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image instance segmentation method, characterized in that, include: Obtain an image to be processed and a query request to be processed, wherein the display content in the image to be processed is a target object, and the query request to be processed includes a location embedding part and a content part, wherein the location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object; A target camouflage instance segmentation model is used to analyze the query request to be processed and the image to be processed to obtain the segmentation result of the target object. The target camouflage instance segmentation model is obtained by optimizing the model parameters of an initial camouflage instance segmentation model based on a first loss and a second loss. The first loss and the second loss are determined based on a first prediction result and a second prediction result. The first prediction result and the second prediction result are obtained by training the initial camouflage instance segmentation model using a training dataset. The first loss is a similarity metric loss, and the second loss is a cross-entropy loss. The first prediction result is the mask prediction result of the instance in the sample image corresponding to the sample query set, and the second prediction result is the position prediction result of the instance in the sample image corresponding to the sample query set. The training dataset includes the sample query set and the sample image.

2. The image instance segmentation method according to claim 1, characterized in that, The image instance segmentation method further includes: The initial masquerading instance segmentation model is used to perform an activity estimation measurement on each sample query in the sample query set to obtain the measurement results; The measurement results are used to determine the number of first sample queries, wherein the number of first sample queries is used to select multiple first sample queries from the sample query set, and the multiple first sample queries are multiple sensitive queries; Based on the first sample query count, cross-attention processing is performed on the multiple first sample queries to obtain the first prediction result.

3. The image instance segmentation method according to claim 2, characterized in that, For each sample query in the sample query set, an activity estimation measurement is performed to obtain the measurement results, including: Obtain the relative entropy and variance of multiple extracted features for each sample query in the sample query set, wherein the relative entropy is used to measure the information content of the multiple extracted features, and the variance is used to represent the dispersion of the multiple extracted features; The measurement result is obtained by performing an active estimation measurement based on the relative entropy and the variance.

4. The image instance segmentation method according to claim 2, characterized in that, Determining the number of the first sample queries using the measurement results includes: Based on the measurement results, determine the number of second sample queries corresponding to each head in the multi-head cross-attention mechanism; The maximum value is selected from the second sample query counts corresponding to multiple heads to obtain the first sample query count.

5. The image instance segmentation method according to claim 2, characterized in that, Based on the first sample query count, multi-head cross-attention processing is performed on the multiple first sample queries to obtain the first prediction result, including: Based on the first number of sample queries, the plurality of first sample queries are selected from the sample query set, and the plurality of second sample queries are discarded, wherein the plurality of second sample queries are multiple dormant queries; The first prediction result is obtained by performing multi-head cross-attention processing on the multiple first sample queries.

6. The image instance segmentation method according to claim 2, characterized in that, The image instance segmentation method further includes: The initial camouflaged instance segmentation model is used to analyze the location embedding part of each sample query in the sample query set to obtain the second prediction result, wherein the location embedding part of each sample query is pre-extracted from the boundary points of the target instance in the sample image.

7. The image instance segmentation method according to claim 1, characterized in that, The method further includes providing a graphical user interface (GUI) via a cloud-native platform, wherein the content displayed by the GUI at least partially comprises an object masquerading instance segmentation scenario, and the method also includes: In response to a first touch operation applied to the graphical user interface, the image to be processed is selected from multiple candidate images; In response to a second touch operation applied to the graphical user interface, the target masquerading instance segmentation model is selected from multiple candidate instance segmentation models; In response to a third touch operation applied to the graphical user interface, the target camouflage instance segmentation model is used to analyze the query request to be processed and the image to be processed to obtain the segmentation result; The segmentation results are displayed within the graphical user interface.

8. An image instance segmentation method, characterized in that, include: The process involves acquiring an image of a city building complex to be processed and a query request to be processed. The image of the city building complex to be processed contains the target building. The query request to be processed includes a location embedding part and a content part. The location embedding part is used to predict the location information of the target building, and the content part is used to perform target detection on the target building. A target camouflage instance segmentation model is used to analyze the query request to be processed and the image of the city building complex to be processed to obtain the segmentation result corresponding to the target building. The target camouflage instance segmentation model is obtained by optimizing the model parameters of the initial camouflage instance segmentation model based on a first loss and a second loss. The first loss and the second loss are determined based on the first prediction result and the second prediction result. The first prediction result and the second prediction result are obtained by training the initial camouflage instance segmentation model using a training dataset. The first loss is a similarity metric loss, and the second loss is a cross-entropy loss. The first prediction result is the mask prediction result of the instance in the sample image corresponding to the sample query set, and the second prediction result is the position prediction result of the instance in the sample image corresponding to the sample query set. The training dataset includes the sample query set and the sample image.

9. The image instance segmentation method according to claim 8, characterized in that, The method further includes providing a graphical user interface (GUI) via a cloud-native platform, wherein the content displayed by the GUI at least partially comprises a scene segmented by masquerading city buildings, and the method also includes: In response to a first touch operation applied to the graphical user interface, the image of the urban building complex to be processed is selected from multiple candidate urban building complex images; In response to a second touch operation applied to the graphical user interface, the target masquerading instance segmentation model is selected from multiple candidate instance segmentation models; In response to a third touch operation applied to the graphical user interface, the target camouflage instance segmentation model is used to analyze the query request to be processed and the image of the urban building complex to be processed, and the segmentation result is obtained. The segmentation results are displayed within the graphical user interface.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the image instance segmentation method according to any one of claims 1 to 9.

11. An electronic device, characterized in that, include: processor; as well as A memory, connected to the processor, for providing the processor with instructions to perform the following processing steps: Obtain an image to be processed and a query request to be processed, wherein the display content in the image to be processed is a target object, and the query request to be processed includes a location embedding part and a content part, wherein the location embedding part is used to predict the location information of the target object, and the content part is used to perform target detection on the target object; A target camouflage instance segmentation model is used to analyze the query request to be processed and the image to be processed to obtain the segmentation result of the target object. The target camouflage instance segmentation model is obtained by optimizing the model parameters of an initial camouflage instance segmentation model based on a first loss and a second loss. The first loss and the second loss are determined based on a first prediction result and a second prediction result. The first prediction result and the second prediction result are obtained by training the initial camouflage instance segmentation model using a training dataset. The first loss is a similarity metric loss, and the second loss is a cross-entropy loss. The first prediction result is the mask prediction result of the instance in the sample image corresponding to the sample query set, and the second prediction result is the position prediction result of the instance in the sample image corresponding to the sample query set. The training dataset includes the sample query set and the sample image.

Images