Object determination method and apparatus, storage medium, and electronic device

Abstract

The application discloses a kind of object determination method, device, storage medium and electronic equipment, wherein the method comprises: receiving the text information input by target user and reference image;Determine fusion features based on text information and reference image;Determine first visual features based on reference image, wherein the first visual features are the features obtained by enhancing appearance information;Determine second visual features based on reference image and fusion features, wherein the second visual features are the features obtained by enhancing the image of the first object included in reference image;Determine target feature vector based on first visual features, second visual features and fusion features;Determine target object from real-time image based on target feature vector.Through the present application, the problem of inaccurate determination of target object is solved, and the effect of accurately determining the target object is achieved.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method, apparatus, storage medium, and electronic device for determining an object. Background Technology

[0002] In related technologies, the single-modal (text) cues used to identify target objects are often insufficient, especially in scenarios where language is difficult to describe precisely. They are also difficult to accurately understand and adapt to user-defined low-altitude targets, and their generalization ability is insufficient for small samples.

[0003] This indicates that there is a technical problem in the relevant technology of inaccurate identification of the target object.

[0004] There is currently no effective solution to the aforementioned problems in the relevant technologies. Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and electronic device for determining an object, so as to at least solve the technical problem of inaccurate determination of target objects in related technologies.

[0006] According to one aspect of the embodiments of this application, a method for determining an object is provided, comprising: receiving text information input by a target user and a reference image, wherein the text information is used to represent textual description information of a target object to be determined, and the reference image is used to provide appearance information of the target object; determining a fusion feature based on the text information and the reference image; determining a first visual feature based on the reference image, wherein the first visual feature is a feature obtained by enhancing the appearance information; determining a second visual feature based on the reference image and the fusion feature, wherein the second visual feature is a feature obtained by enhancing an image of a first object included in the reference image, and the number of pixels of the first object is less than or equal to a preset number; determining a target feature vector based on the first visual feature, the second visual feature, and the fusion feature; and determining the target object from a real-time image based on the target feature vector.

[0007] In an exemplary embodiment, determining a first visual feature based on the reference image includes: performing a multi-scale feature extraction operation on the reference image to obtain multiple hierarchical features; determining a first hierarchical feature included among the multiple hierarchical features, wherein the first hierarchical feature is used to describe the visual features of the reference image; and determining the first visual feature based on the first hierarchical feature.

[0008] In an exemplary embodiment, determining the first visual feature based on the first hierarchical feature includes: extracting frequency domain features of the first hierarchical feature; reducing the dimensionality of the frequency domain feature to obtain a first feature; generating a first weight based on the first feature; fusing the frequency domain feature with the first weight to obtain a second feature; recombining the second feature to obtain a third feature; and fusing the first hierarchical feature with the third feature to obtain the first visual feature.

[0009] In an exemplary embodiment, determining a second visual feature based on the reference image and fused features includes: performing multi-scale feature extraction on the reference image to obtain multiple hierarchical features; determining a first-level feature, a second-level feature, and a third-level feature included in the multiple hierarchical features, wherein the first-level feature and the second-level feature are used to describe the visual features of the reference image, and the third-level feature is used to extract semantic information in the reference image; determining a first sub-visual feature based on the first-level feature, the third-level feature, and the fused features; determining a second sub-visual feature based on the second-level feature, the third-level feature, and the fused features; and determining the second visual feature based on the first sub-visual feature and the second sub-visual feature.

[0010] In an exemplary embodiment, determining a first sub-visual feature based on the first-level feature, the third-level feature, and the fused feature includes: extracting contextual features of the first-level feature to obtain a fourth feature; fusing the fourth feature with the third-level feature to obtain a fifth feature; determining a second weight based on the fused feature; fusing the fifth feature with the second weight to obtain a sixth feature; performing feature extraction on the sixth feature to obtain a seventh feature; extracting gradient features of the sixth feature to obtain an eighth feature; and concatenating the eighth feature, the seventh feature, and the first-level feature to obtain the first sub-visual feature.

[0011] In one exemplary embodiment, determining fusion features based on the text information and the reference image includes: determining text features based on the text information; determining image features based on the reference image; and determining the fusion features based on the text features and the image features.

[0012] In an exemplary embodiment, determining the fusion feature based on the text features and the image features includes: performing a channel concatenation operation on the text features and the image features to obtain a concatenated feature; determining a third weight corresponding to the text features and a fourth weight corresponding to the image features; performing a mapping operation on the image features to obtain a ninth feature, the mapping operation being used to align the first semantic space of the image features with the second semantic space of the text features; weighting the third weight with the text features to obtain a first sub-fusion feature; weighting the fourth weight with the ninth feature to obtain a second sub-fusion feature; and determining the sum of the first sub-fusion feature and the second sub-fusion feature as the fusion feature.

[0013] In one exemplary embodiment, determining the target object from a real-time image based on the target feature vector includes: determining target query features of the real-time image; determining a similarity score between the target query features and the target feature vector; and determining the target object from the real-time image based on the similarity score.

[0014] According to another aspect of the embodiments of this application, an object determination apparatus is also provided, comprising: a receiving module, configured to receive text information input by a target user and a reference image, wherein the text information is used to represent textual description information of a target object to be determined, and the reference image is used to provide appearance information of the target object; a first determining module, configured to determine a fusion feature based on the text information and the reference image; a second determining module, configured to determine a first visual feature based on the reference image, wherein the first visual feature is a feature obtained by enhancing the appearance information; a third determining module, configured to determine a second visual feature based on the reference image and the fusion feature, wherein the second visual feature is a feature obtained by enhancing an image of a first object included in the reference image, and the number of pixels of the first object is less than or equal to a preset number; a fourth determining module, configured to determine a target feature vector based on the first visual feature, the second visual feature, and the fusion feature; and a fifth determining module, configured to determine the target object from a real-time image based on the target feature vector.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed by a processor.

[0016] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the steps in any of the method embodiments described above.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to perform the steps of any of the above method embodiments through the computer program.

[0018] This application allows for the initial receipt of text information and a reference image input by a target user. Fusion features can be determined using the text information and reference image. The text information represents the textual description of the target object to be identified, while the reference image provides the object's appearance information. A first visual feature, enhanced by augmenting the appearance information, can be determined using the reference image. A second visual feature, enhanced by augmenting an image of a first object in the reference image where the number of pixels is less than or equal to a preset number, can also be determined using the reference image and fusion features. Finally, a target feature vector can be determined using the first visual feature, the second visual feature, and the fusion feature. This target object can then be identified from the real-time image using the target feature vector. Since the target feature vector can be determined based on the first visual feature enhanced by augmenting the appearance information, the second visual feature enhanced for small targets, and the fusion feature determined by the text description and reference image, and the target object can be identified using the target feature vector, the model can quickly establish appearance matching using the reference image when facing novel low-altitude targets not seen in the training set, combined with textual semantic constraints. This approach not only ensures that the model avoids losing context due to excessive focus on details, but also prevents inaccurate target object identification caused by focusing solely on semantics and ignoring appearance differences. Furthermore, it enables high-precision target detection in zero-shot or low-sample scenarios. Therefore, it solves the problem of inaccurate target object identification, achieving accurate target object identification. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of a method for determining an object according to an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating an optional method for determining an object according to an embodiment of this application;

[0021] Figure 3 This is a flowchart illustrating the method for determining the object in this optional example;

[0022] Figure 4 This is a schematic diagram of the structure of the object determination system according to an embodiment of this application;

[0023] Figure 5 This is a structural block diagram of an optional object determining device according to an embodiment of this application;

[0024] Figure 6 This is a computer system architecture block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] According to one aspect of the embodiments of this application, a method for determining an object is provided. Optionally, in this embodiment, the above-described method for determining an object may be applied, but is not limited to, to objects such as... Figure 1 The hardware environment shown includes terminal device 102 and server 104. Server 104 can be connected to terminal device 102 via a network and can be used to provide services (e.g., application services, etc.) to terminal device 102 or clients installed on terminal device 102. A database can be set up on server 104 or independently of server 104 to provide data storage services for server 104.

[0028] The aforementioned network may include, but is not limited to, at least one of the following: wired network and wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network (WAN), metropolitan area network (MAN), and local area network (LAN). The aforementioned wireless network may include, but is not limited to, at least one of the following: Wireless Fidelity (WIFI) and Bluetooth. Terminal device 102 may be, but is not limited to, a personal computer (PC), mobile phone, tablet computer, etc. Server 104 may be, but is not limited to, a cloud server, server cluster, or other server types.

[0029] The object determination method of this application embodiment can be executed by server 104, terminal device 102, or jointly by server 104 and terminal device 102. Alternatively, the object determination method of this application embodiment can be executed by a client installed on terminal device 102.

[0030] Figure 2 This is a flowchart illustrating an optional object determination method according to an embodiment of this application, such as... Figure 2 As shown, the process of this method may include the following steps:

[0031] Step S202: Receive text information and a reference image input by the target user, wherein the text information is used to represent the text description information of the target object to be determined, and the reference image is used to provide the appearance information of the target object;

[0032] The object determination method in this embodiment can be applied to technologies such as low-altitude visual perception, multimodal large models, image processing, and AI training and inference integrated servers. In related technologies, existing low-altitude visual models typically employ a training and inference separation architecture. Training is usually performed in the cloud or on high-performance servers. After training, model files need to be manually exported and then distributed and deployed to edge inference devices. This prevents the formation of a training-inference closed loop within a single integrated machine, resulting in lagging model iteration and low collaborative efficiency. Furthermore, existing low-altitude detection models only support single-modal cues, cannot achieve combined image and text cues, struggle to accurately understand and adapt to user-defined low-altitude targets, have insufficient small-sample generalization ability, and cannot perform hardware collaborative optimization for low-altitude aerial image distortion, small targets, and motion blur. They also lack a collaborative scheduling mechanism for heterogeneous hardware (training unit + inference unit), leading to resource conflicts between training and inference tasks and hindering efficient and stable collaborative operation.

[0033] To at least partially solve the above-mentioned technical problems, in this embodiment, a heterogeneous hardware integrated architecture of training unit and inference unit can be adopted, with an internal high-speed bus and a collaborative scheduling engine, to realize automatic closed-loop training, inference and updating within a single machine. Compared with the existing training and inference separation architecture (which requires manual distribution of models to edge devices) and multi-model ring integration (without an automatic update mechanism), the entire cycle from the collection of new target data to the model taking effect at the inference end can be shortened to less than 4 hours, and the iteration efficiency is improved by more than 10 times.

[0034] In at least one embodiment, the system can acquire raw aerial image data by connecting to various low-altitude visual sensing devices, including but not limited to UAV-borne visual devices, low-altitude cameras, or airborne visual devices. UAV-borne visual devices can receive video streams or single-frame images transmitted from the UAV in real time via Wi-Fi, 4G / 5G, or a dedicated data transmission link. Low-altitude cameras can connect to fixed high-definition cameras deployed in a low-altitude monitoring network to continuously acquire monitoring images of specific areas. Airborne visual devices can upload manually captured or selected low-altitude images via a mobile app or ground control station. Target users can interactively annotate the acquired images using an online annotation system. Specifically, target users can draw rectangles on the preview image to define objects of interest. The system will automatically record the coordinates of each bounding box to form the spatial location information of the target. Target objects can include, but are not limited to, vehicles, pedestrians, boats, animals, etc. For each labeled target bounding box, the user can input its corresponding text description information (i.e., the text information mentioned above). The text description information can include, but is not limited to, the target object's name (e.g., police car, ambulance, illegal building, etc.), attribute features (e.g., color, size, and material, etc.), and scene semantics (e.g., moving, stationary, illegally dumped, etc.). In addition to the text description information, the user can also input a target example image (i.e., the reference image mentioned above) as a visual prompt to provide prior information about the target object's appearance. Since low-altitude targets often have unique textures, paint schemes, or shape features, relying solely on text descriptions may not be enough to accurately distinguish them. Therefore, providing example images allows the model to learn the specific visual features of the target object (such as specific body patterns, color combinations, etc.).

[0035] Step S204: Determine fusion features based on the text information and the reference image;

[0036] In at least one embodiment, the system can uniformly organize the annotated original image, target bounding box annotations, text prompts (i.e., the aforementioned text information), and example image prompts (i.e., the aforementioned reference images) into a low-altitude dedicated image-text joint cue dataset. The text prompts can be used to describe target semantic information, such as category names, attribute features, and scene semantics; the image prompts can be used to provide prior information about the target's appearance, such as example images with specific textures, colors, and shapes. Specifically, the data format of the low-altitude dedicated image-text joint cue dataset is: {Original Image I, Target Bounding Box Set} Text prompt collection Visual cue set Each target box b i Each is associated with a corresponding text prompt t i And a visual cue v i .

[0037] In at least one embodiment, the text prompt and the image prompt can be input into the image-text joint prompt encoding module to generate joint prompt features (i.e., the aforementioned fusion features). The image-text joint prompt encoding model can directly perform a weighted summation of the vector corresponding to the text prompt and the vector corresponding to the image prompt to generate fusion features. Alternatively, it can perform data processing on the vector corresponding to the text prompt and the vector corresponding to the image prompt respectively, and then perform a weighted summation of the processed vectors to generate fusion.

[0038] Step S206: Determine a first visual feature based on the reference image, wherein the first visual feature is a feature obtained by enhancing the appearance information;

[0039] In at least one embodiment, images captured by low-altitude drones or cameras differ significantly from conventional ground images, typically exhibiting three main problems: motion blur and vibration noise, loss of detail for distant small targets, and compression artifacts and uneven illumination. Specifically, motion blur and vibration noise can be understood as the high-frequency vibrations caused by airflow during drone flight, resulting in motion blur in the image. This blur manifests as a smooth transition in pixel values, making it difficult for CNNs to distinguish between object edges and blurry noise. Loss of detail for distant small targets can be immediately apparent because targets captured at low altitudes (such as faces, license plates, and illegally parked vehicles) occupy a very small portion of the image (often less than 32x32 pixels). After multiple downsampling layers, subtle texture information in shallow features is easily lost, leading to failure in small target detection. Compression artifacts and uneven illumination can be understood as follows: for real-time transmission, aerial videos are typically compressed using JPEG / H.264, producing blockiness and ringing effects. Additionally, drastic changes in low-altitude lighting (such as under trees or in sunlight) result in extremely low local contrast. Therefore, to address the aforementioned issues and enhance the stability of blurred targets, small targets, and compressed images in low-altitude scenes, a frequency domain enhancement branch of DWT can be introduced to process the reference image. This branch can explicitly separate and enhance the first visual features containing rich details of small target appearance from a frequency perspective, compensating for the shortcomings of traditional spatial CNNs in texture and edge capture. Besides DWT, other methods such as FFT (Fast Fourier Transform), wavelet transform, or Gabor filter banks can also be used. The number of frequency domain components and the dimension of attention weights can be adjusted during application.

[0040] Step S208: Determine a second visual feature based on the reference image and the fusion features, wherein the second visual feature is a feature obtained by enhancing the image of the first object included in the reference image, and the number of pixels of the first object is less than or equal to a preset number.

[0041] In at least one embodiment, for low-altitude, long-range small targets, since the flight altitude of drones is typically 50-120 meters, target objects (such as pedestrians, vehicles, illegally parked boats, and small drones) often occupy a very small pixel area. The target pixel area is usually less than 32×32, many are within 20×20, or even 10×10. For example, an adult in an image may only have 10x10 to 30x30 pixels. Therefore, the first object in the reference image with a number of pixels less than or equal to a preset number can be enhanced. Specifically, a small target enhancement module can be introduced to enhance the reference image and fused features. The small target enhancement module can generate second visual features by expanding the receptive field, fusing multi-scale semantics, using prompts to guide attention, and enhancing edge textures. This can solve the problems of weak small target features and susceptibility to noise, thereby significantly improving the detection accuracy and recall rate in low-altitude scenes.

[0042] Step S210: Determine the target feature vector based on the first visual feature, the second visual feature, and the fused feature;

[0043] In at least one embodiment, after determining the first visual feature, the second visual feature, and the fused feature using the reference image respectively, the three can be fused. Specifically, since the first visual feature is a feature obtained by enhancing appearance information, it can significantly enhance the structural integrity and edge sharpness of the target; since the second visual feature enhances small target objects, it can ensure that small targets have sufficient response intensity in the feature map, preventing missed detection due to low resolution; since the fused feature is a fusion of the reference image and text information, including user-defined semantic information (text) and appearance prior (image), fusing the three is equivalent to adding a semantic filter to the feature map. Only feature regions that have both clear edges and high-resolution details, and match the semantic prompt, will be activated. That is, when the image is blurry, the first visual feature can provide stable edge information; when the target is extremely small, the second visual feature can provide magnified detail response; and when the semantics are ambiguous, the visual prompt in the fused feature can provide appearance reference. The fusion method can include, but is not limited to, weighted fusion, and can also use cross-attention mechanism or gating fusion mechanism, with the fused feature as the Query and the concatenated feature of the first visual feature and the second visual feature as the Key and Value.

[0044] Step S212: Determine the target object from the real-time image based on the target feature vector.

[0045] In at least one embodiment, to overcome the problem that traditional RT-DERT detectors can only identify predefined fixed categories (closed-set detection), this embodiment modifies the output head of the detection network to open-set. Specifically, the traditional RT-DERT detection head contains a fully connected layer that maps the target query features output by the decoder to C-dimensional class confidence scores, where C is the number of categories predefined during training. Its classification logic is based on the Softmax function, assuming that all possible target categories are included in the training set. When faced with new categories not defined in the training set (such as new types of drones or specific illegal buildings), the traditional model cannot generate an effective response and can only misclassify them as the most similar known category or completely miss detection. In this embodiment, however, the correlation strength between the target feature vector and the target query features can be calculated instead of the traditional Softmax classification. Since OVMH does not rely on a fixed class weight matrix, this embodiment does not require model fine-tuning or retraining and can identify any new user-defined target in real time.

[0046] This application allows for the initial receipt of text information and a reference image input by a target user. Fusion features can be determined using the text information and reference image. The text information represents the textual description of the target object to be identified, while the reference image provides the object's appearance information. A first visual feature, enhanced by augmenting the appearance information, can be determined using the reference image. A second visual feature, enhanced by augmenting an image of a first object in the reference image where the number of pixels is less than or equal to a preset number, can also be determined using the reference image and fusion features. Finally, a target feature vector can be determined using the first visual feature, the second visual feature, and the fusion feature. This target object can then be identified from the real-time image using the target feature vector. Since the target feature vector can be determined based on the first visual feature enhanced by augmenting the appearance information, the second visual feature enhanced for small targets, and the fusion feature determined by the text description and reference image, and the target object can be identified using the target feature vector, the model can quickly establish appearance matching using the reference image when facing novel low-altitude targets not seen in the training set, combined with textual semantic constraints. This approach not only ensures that the model avoids losing context due to excessive focus on details, but also prevents inaccurate target object identification caused by focusing solely on semantics and ignoring appearance differences. Furthermore, it enables high-precision target detection in zero-shot or low-sample scenarios. Therefore, it solves the problem of inaccurate target object identification, achieving accurate target object identification.

[0047] Optionally, the entity performing the above steps may be a terminal, a server, a client, or other devices with similar processing capabilities, but is not limited to these.

[0048] In an exemplary embodiment, determining a first visual feature based on the reference image includes: performing a multi-scale feature extraction operation on the reference image to obtain multiple hierarchical features; determining a first hierarchical feature included among the multiple hierarchical features, wherein the first hierarchical feature is used to describe the visual features of the reference image; and determining the first visual feature based on the first hierarchical feature.

[0049] In at least one embodiment, to enhance the stability of blurred targets, small targets, and compressed images in aerial scenes, a DWT frequency domain enhancement branch can be introduced. Specifically, DINOv3-Tiny can be used as the backbone network to extract multi-scale features from the user-uploaded reference image (inference resolution 800×800). The backbone network can perform forward propagation on the input reference image to extract a multi-scale feature pyramid containing different semantic levels and spatial resolutions. In this embodiment, the backbone can output four levels of feature maps, denoted as P2, P3, P4, and P5, with spatial resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. Since target objects (such as drones, vehicles, pedestrians, etc.) in low-altitude aerial images are usually small targets with low pixel ratios, and the P2 layer features (i.e., the first-level features mentioned above) retain a high spatial resolution, they can effectively preserve the texture details, edge contours, and fine structural information of the target object. Therefore, the first-level features can be used to determine the first visual features.

[0050] In at least one embodiment, while the P5 feature possesses the strongest semantic generalization capability, its spatial resolution is extremely low (only 1 / 32 of the original image). In low-altitude small target detection scenarios, the P5 layer features often result in information loss or severe blurring of small targets due to their low resolution, and the computational cost of this layer is also significant. Therefore, to improve the overall inference speed of the system and reduce computational consumption, the P5 feature can be omitted from the encoder structure; only P2, P3, and P4 are fed into the encoder to improve inference speed. In this configuration, each layer's features are uniformly mapped to 256 channels via a 1×1 convolution.

[0051] In at least one embodiment, the standard EWC algorithm incurs high storage overhead for calculating the Fisher information matrix of the entire network (approximately 2GB per task for RT-DETR) and provides equal protection to all layers, limiting its adaptability to new targets. Therefore, L-EWC can be improved. Besides L-EWC, other methods such as LwF (Learning without Forgetting, knowledge distillation constraint), MAS (Memory Aware Synapses, online parameter importance estimation), or Replay-based methods (storing representative samples from previous tasks for joint training) can also be used. Firstly, hierarchical Fisher computation can be implemented: specifically, the Fisher information matrix can be calculated only for the parameters of the detection head (OVMH) and the shallow feature extraction layers P2 / P3. The backbone network (DINOv3 tiny) has fixed parameters and does not participate in incremental updates, thus reducing the number of parameters by 85% and computational overhead by 90%. The diagonal elements of the Fisher information matrix can be calculated using the following formula: Where D represents the distribution of historical task data, and L represents the historical task loss function. Secondly, the adaptive importance threshold can be determined: specifically, traditional EWC uses a globally fixed regularization coefficient λ, while in this embodiment, a parameter-level adaptive coefficient can be introduced. ,in, This can be understood as the mean of the Fisher values ​​in the current layer. It can prevent division by zero. It can take the value 10. 4 -10 6 It can dynamically adjust based on task similarity. Stronger protection can be applied to parameters significantly above the mean (key parameters from historical tasks), while parameters close to the mean can be allowed larger update increments. Furthermore, a progressive forgetting boundary can be set; specifically, an importance judgment threshold can be set. ( (where θ is the standard deviation). Parameters with Fisher values ​​greater than the threshold θ can be frozen and prevented from updating. Fisher values ​​are below 0.5. The parameters in the θ interval can be updated incrementally, provided the Fisher value is less than or equal to 0.5. The parameters can be updated freely and iteratively without constraints. Therefore, the final incremental training total loss function is: ,in, This can be understood as the set of updatable parameters of the Kth layer network. This can be understood as the convergence of optimal network parameters for historical tasks. By employing hierarchical adaptive EWC (L-EWC) incremental learning, utilizing the Fisher information matrix to evaluate parameter importance, updating only non-critical parameters, and adaptively adjusting the regularization strength, this embodiment can achieve an accuracy retention rate of ≥95% for the original target after training with a new target, avoiding catastrophic forgetting.

[0052] In this embodiment, by extracting multi-level features from the reference image and selecting specific first-level features as the final visual features, the richness of feature representation and computational efficiency can be effectively balanced. Multi-scale feature extraction ensures that the details and semantic information of the reference image at different resolutions are preserved, and enhances the adaptability of features to the diversity of target morphology in low-altitude scenes.

[0053] In an exemplary embodiment, determining the first visual feature based on the first hierarchical feature includes: extracting frequency domain features of the first hierarchical feature; reducing the dimensionality of the frequency domain feature to obtain a first feature; generating a first weight based on the first feature; fusing the frequency domain feature with the first weight to obtain a second feature; recombining the second feature to obtain a third feature; and fusing the first hierarchical feature with the third feature to obtain the first visual feature.

[0054] In at least one embodiment, firstly, frequency domain attention calculation can be performed: Frequency domain features of the P2 layer visual features (i.e., the first-level features mentioned above) can be extracted. Specifically, a two-level discrete wavelet transform (Haae wavelet) can be performed on the first-level features to obtain four frequency domain components (i.e., the frequency domain features mentioned above): LL, LH, HL, and HH. The feature dimension is H / 4×W / 4×C, where LL (low-low frequency) can be used to preserve the main semantic information, and LH (low-high frequency), HL (high-low frequency), and HH (high-high frequency) can be used to enhance edge and texture features. Each frequency domain feature is obtained by decomposing the P2 layer features through a two-level discrete wavelet transform (DWT2). Dimensionality reduction can be performed on each of these features. Specifically, a global average pooling operation can be performed on each frequency domain feature to obtain a C-dimensional channel dimension statistical feature (i.e., the first feature mentioned above). Then, for each first feature, a first weight corresponding to each frequency domain feature can be generated. Specifically, a two-layer MLP channel transform network can be constructed. The first layer weight dimension W1 is C×C / 16, and the second layer weight dimension W2 is C / 16×C. After passing through ReLU activation and Sigmoid activation, C-dimensional channel attention weights are finally generated. (i.e., the first feature mentioned above), the formula is as follows: Where b1 is the bias term of the first fully connected layer and b2 is the bias term of the second fully connected layer.

[0055] In at least one embodiment, after determining the first weight, channel-level recalibration can be performed: the frequency domain features can be fused with the first weight to obtain the second feature. Specifically, channel-level recalibration can be performed on the frequency domain features using channel attention weights, and the frequency domain features can be weighted and enhanced channel by channel using the following formula to generate the second feature. In this context, ⊙ can be understood as a channel-by-channel multiplication operation. This is the first weight corresponding to the frequency domain feature. This refers to frequency domain features, which allows for the enhancement of important frequency domain channels and the suppression of ineffective features.

[0056] In at least one embodiment, the second feature can also be reconstructed using inverse wavelet transform: that is, the second feature can be recombined to obtain the third feature. Specifically, a second-order inverse wavelet transform can be performed on the four weighted and enhanced frequency domain components (i.e., the aforementioned second feature) LL', LH', HL', and HH' to complete the frequency domain feature reconstruction and obtain the frequency domain enhanced feature (i.e., the aforementioned third feature). The feature dimension is Finally, spatial-frequency domain fusion can be performed, which means fusing the first-level features with the third-level features to obtain the first visual feature. Specifically, the original spatial features and frequency-domain enhanced features can be fused using the following formula to construct the spatial-frequency domain adaptive fusion output feature: ,in, The original P2 layer spatial features (i.e., the first-level features mentioned above) are preserved by bypassing connections to retain basic spatial information. α It can be understood as a learnable fusion coefficient, constrained to the 0-1 range by the Sigmoid function. The initial value can be set to 0.5, which can adaptively adjust the fusion ratio of spatial features and frequency domain enhanced features to achieve optimal feature enhancement.

[0057] In this embodiment, by converting spatial domain features to the frequency domain and extracting frequency domain features, high-frequency details such as edges and textures of images can be captured more sensitively. Through dimensionality reduction, weight generation, and fusion and recombination operations, adaptive complementarity and recalibration of frequency domain enhancement features and original spatial domain features are achieved. This not only suppresses background noise interference but also strengthens the discriminative features of key targets, thereby significantly improving the accuracy and robustness of subsequent image-text joint prompting modules in target perception and matching in complex low-altitude scenes.

[0058] In an exemplary embodiment, determining a second visual feature based on the reference image and fused features includes: performing multi-scale feature extraction on the reference image to obtain multiple hierarchical features; determining a first-level feature, a second-level feature, and a third-level feature included in the multiple hierarchical features, wherein the first-level feature and the second-level feature are used to describe the visual features of the reference image, and the third-level feature is used to extract semantic information in the reference image; determining a first sub-visual feature based on the first-level feature, the third-level feature, and the fused features; determining a second sub-visual feature based on the second-level feature, the third-level feature, and the fused features; and determining the second visual feature based on the first sub-visual feature and the second sub-visual feature.

[0059] In at least one embodiment, DINOv3-Tiny can be used as the backbone network to perform multi-scale feature extraction on the user-uploaded reference image (inference resolution 800×800). The backbone network can perform forward propagation on the input reference image to extract a multi-scale feature pyramid containing different semantic levels and spatial resolutions. In this embodiment, the backbone can output feature maps of four levels, denoted as P2 (i.e., the first level feature mentioned above), P3 (i.e., the second level feature mentioned above), P4 (i.e., the third product feature mentioned above), and P5, respectively, with spatial resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. Among them, shallow features P2 and P3 can be used to preserve texture and edge information, and deep feature P4 can be used to extract semantic information. Because small targets at low altitudes and long distances (drones fly at altitudes of 50-120 meters, and the target pixel area is usually much smaller than 32×32, many are within 20×20, or even within 10×10), in order to enhance small targets, in this embodiment, a Small Object Enhancement Module (SOEM) can be added to the P2 layer and the P3 shallow feature layer respectively. Specifically, the small target enhancement module can determine the first sub-visual feature through the first-level feature, the third-level feature, and the fused feature, and determine the second sub-visual feature through the second-level feature, the third-level feature, and the fused feature. Finally, the first sub-visual feature and the second sub-visual feature can be fused to obtain the second visual feature. The fusion method can include, but is not limited to, weighted fusion, additive fusion, etc., and this application does not limit it.

[0060] This embodiment utilizes shallow features to preserve high-resolution spatial information such as edge texture of small targets, combines deep features to provide strong semantic context to suppress background interference, and introduces fused features to guide cross-modal semantic alignment. Through the synergistic complementarity of multiple sub-features, it effectively solves the feature ambiguity problem caused by large target scale changes and complex backgrounds in low-altitude scenes, thereby significantly improving the model's recognition accuracy, generalization ability, and robustness of feature representation for custom low-altitude targets.

[0061] In an exemplary embodiment, determining a first sub-visual feature based on the first-level feature, the third-level feature, and the fused feature includes: extracting contextual features of the first-level feature to obtain a fourth feature; fusing the fourth feature with the third-level feature to obtain a fifth feature; determining a second weight based on the fused feature; fusing the fifth feature with the second weight to obtain a sixth feature; performing feature extraction on the sixth feature to obtain a seventh feature; extracting gradient features of the sixth feature to obtain an eighth feature; and concatenating the eighth feature, the seventh feature, and the first-level feature to obtain the first sub-visual feature.

[0062] In at least one embodiment, feature enhancement using a small target enhancement module can include the following process: First, a dilated convolution branch can be executed: the contextual features of the first-level features can be extracted to obtain the fourth feature. Specifically, two parallel 3×3 dilated convolutional layers with dilationrate=2 and dilationrate=4 can be used to process the first-level features respectively, and the two features obtained are concatenated to obtain the fourth feature. The two dilated convolutional layers can expand the effective receptive field of the first-level features to 7×7 and 9×9 respectively, which can be used to capture a wider range of contextual information without sacrificing spatial resolution. Second, high-resolution cross-layer fusion can be achieved: that is, the fourth feature and the third-level features can be fused to obtain the fifth feature. Specifically, we can first obtain the P4 layer features (i.e., the third-level features mentioned above) output from the visual backbone network. Then, we upsample these third-level features using bilinear interpolation or transposed convolution to increase their spatial resolution from the current layer to 1 / 4 the resolution of the P2 layer. Afterward, we can fuse the fourth feature with the upsampled P4 features element-wise or through channel concatenation to generate a fifth feature that preserves high-resolution details and introduces deeper semantics. Furthermore, we can perform Prompt-guided channel attention: that is, we can determine the second weight based on the fused features, and then fuse the fifth feature with the second weight to obtain the sixth feature. Specifically, we can combine the cue features... The aforementioned fusion features are globally averaged and processed using a two-layer MLP (256→64→C, where C is the number of channels in the current layer, with Sigmoid activation) to generate channel weights (i.e., the second weights mentioned above). This means the dimension 256 can be compressed to 64 using the first fully connected layer, and the dimension is restored to C using the second fully connected layer. Then, the output is normalized using the Sigmoid activation function, limiting the weight values ​​of each of the C channels to the range (0-1). Finally, the second weights and the fifth feature are recalibrated at the channel level and fused through channel-wise multiplication to obtain the sixth feature, enhancing the channel responses relevant to the cue target. Furthermore, fine-grained texture enhancement can be implemented to improve the edge and texture features of small targets: feature extraction is performed on the sixth feature to obtain the seventh feature; the gradient features of the sixth feature are extracted to obtain the eighth feature; and the eighth, seventh, and first-level features are concatenated to obtain the first sub-visual feature. Specifically, the sixth feature can be subjected to 3×3 depthwise separable convolution (channel-wise convolution + 1×1 point convolution) to generate the seventh feature and a Sobel edge detection kernel (horizontal / vertical direction). The Sobel operator can then be used to calculate the gradients in the horizontal and vertical directions, and these gradients can be concatenated to generate the eighth feature. Finally, the eighth and seventh features output by the SOME module can be residually concatenated with the original P2 layer features (i.e., the first-level features mentioned above) to prevent gradient vanishing and preserve the original information flow, thus obtaining the first sub-visual feature. ,in, This refers to the first-level features. These are the seventh and eighth features mentioned above. The process of determining the second sub-visual feature through the second-level features, the third-level features, and the fusion features is the same as the process of determining the first sub-visual feature, and will not be elaborated here.

[0063] In this embodiment, the semantic information of the third-level features guides the extraction of contextual features of the first-level features, thereby achieving semantic-guided spatial detail enhancement. Subsequently, the intermediate features are adaptively recalibrated by dynamic weights generated from fused features, suppressing irrelevant background noise. Furthermore, gradient features are introduced to enhance the sharpness and structural information of target edges, compensating for the edge smoothing loss that may be caused by traditional convolution operations. Finally, by multi-dimensionally splicing deep semantics (seventh feature), structural details (eighth feature), and original high-resolution spatial information (first-level features), a composite feature representation containing rich semantics, strong edges, and original spatial information is formed, thereby significantly enhancing the model's positioning accuracy and recognition robustness in low-altitude small target detection.

[0064] In one exemplary embodiment, determining fusion features based on the text information and the reference image includes: determining text features based on the text information; determining image features based on the reference image; and determining the fusion features based on the text features and the image features.

[0065] In at least one embodiment, MobileCLIP-S0 (embedding dimension 512) can be used as the text encoder to ensure both performance and speed. Specifically: the user-input target name, attribute description, and scene semantic information (i.e., the aforementioned text information) can first be processed by a Tokenizer, with a maximum sequence length of 77. After padding to a fixed length (default fixed length 256), it can be input into the text Transformer encoder. The text encoder outputs 512-dimensional semantic features, which are then compressed to 256 dimensions through two linear mapping layers (512→384→256, each followed by LayerNorm and GELU activation) to reduce the computational load of edge-side inference. The encoded text feature embedding... (256 dimensions) can be used to represent target semantic information. In addition to the MobileCLIP-S0 text encoder, other text encoders can be selected, such as CLIPViT-B / 32 (86M parameters), SigLIP-SO400M (using sigmoid loss, more suitable for large-scale training), or a self-developed lightweight text encoder (based on a 4-layer Transformer-XL structure, with <<8M parameters). In specific use, the input dimension of the linear mapping layer can be adjusted to match the output dimension of the encoder.

[0066] In at least one embodiment, DINOv3-Tiny can be used as the backbone network to extract multi-scale features from the user-uploaded reference image (inference resolution 800×800). The backbone outputs four feature layers, P2, P3, P4, and P5, with spatial resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. Shallow features P2 and P3 are used to preserve texture and edge information, while deep feature P4 is used to extract semantic information. Subsequently, global average pooling and concatenation can be performed on the multi-scale visual features to obtain the visual... Finally, we can assess the visual... Global averaging aggregation is performed to obtain the single-scale visual mean features (i.e., the image features mentioned above). In addition to DINOv3-Tiny, encoders such as ViT-Tiny / 16 (5.7M parameters, optimal speed), ConvNeXt-Tiny (convolutional structure, better edge compatibility), or EfficientNet-B0 (pre-trained on ImageNet, suitable for low-resource scenarios) can also be used. In practical applications, the number of feature pyramid layers and channel mapping strategies can be adjusted.

[0067] In at least one embodiment, after determining the text features and image features, a Dynamic Gating Fusion (DGF) module can be used to fuse the text features and image features to complete cross-modal adaptive fusion.

[0068] This embodiment extracts text features to capture the semantic description of user-defined targets (such as category names, attributes, or scene information), while extracting image features to preserve the appearance, texture, and structural priors of reference examples. Through deep fusion of the two, the model can adaptively balance the importance of text semantic guidance and visual appearance matching, effectively solving the limitations of a single modality in complex low-altitude scenes (such as variable target appearances or ambiguous descriptions), and significantly improving the zero-shot generalization ability and matching accuracy of image-text joint prompts in open-set target detection.

[0069] In an exemplary embodiment, determining the fusion feature based on the text features and the image features includes: concatenating the text features and the image features through channels to obtain concatenated features; determining a third weight corresponding to the text features and a fourth weight corresponding to the image features; performing a mapping operation on the image features to obtain a ninth feature, wherein the mapping operation is used to align the first semantic space of the image features with the second semantic space of the text features; weighting the third weight with the text features to obtain a first sub-fusion feature; weighting the fourth weight with the ninth feature to obtain a second sub-fusion feature; and determining the sum of the first sub-fusion feature and the second sub-fusion feature as the fusion feature.

[0070] In at least one embodiment, the text features and the mean visual features (i.e., the image features mentioned above) can be concatenated first to obtain concatenated features, and then adaptive fusion weights can be calculated through gated branches: Where [·;·] represents the feature concatenation operation. Here are the learnable weights and bias parameters, σ is the sigmoid activation function, and the output is a two-branch weight vector. These correspond to the text modality weights (i.e., the third weight mentioned above) and the visual modality weights (i.e., the fourth weight mentioned above), respectively. Finally, Softmax normalization is used to achieve mutual exclusion constraints on the weights, satisfying... .

[0071] In at least one embodiment, to ensure alignment of visual features with text features in terms of dimension and feature distribution, a linear projection transformation operation can be performed on the aggregated image features to generate a ninth feature. , where the projection matrix These are learnable parameters. Finally, the dual-modal features can be weighted and fused based on normalized gating weights to obtain the final joint cue features (i.e., the fused features mentioned above): Output fusion features The dimension remains consistent with that of the single-modal feature.

[0072] In at least one embodiment, weights can be dynamically assigned based on the quality of the input modal information: when the input consists only of text descriptions and has no example images, visual weights can be automatically constrained. Text weight The detection relies entirely on text prompts; when the text description is semantically ambiguous and the visual image information is clear and reliable, the gating network can autonomously learn to improve the visual weights. In typical scenarios, the visual weights... It prioritizes relying on prior visual information. Through end-to-end training, it automatically learns joint cue features without requiring manual setting of rules or thresholds. Input the RT-DETR detection head as a global query guide condition to constrain the model to complete accurate target detection.

[0073] This embodiment achieves a combined text and image prompting mechanism by fusing text encoding, image encoding, and dynamic gating, enabling zero-shot / few-shot recognition of user-defined low-altitude targets. Compared to existing technologies that only support text prompts (requiring large language model conversion) and only support unimodal prompts, this embodiment supports both text and visual bimodal input. For targets that are difficult to describe precisely with language (such as building roofs with specific textures), example images can be used for more accurate matching.

[0074] In one exemplary embodiment, determining the target object from a real-time image based on the target feature vector includes: determining target query features of the real-time image; determining a similarity score between the target query features and the target feature vector; and determining the target object from the real-time image based on the similarity score.

[0075] In at least one embodiment, traditional RT-DETR uses a fixed-category classifier head with a C-dimensional softmax layer, where C can be understood as the predefined number of categories, capable of only training the recognition of preset category targets. In this embodiment, the original closed-set classifier head is replaced with a cue-guided open-set matching detection head (OVMH). Besides multi-head cross-attention, dynamic convolution, cosine similarity matching, or bidirectional matching networks can also be used. Specifically, the input decoder outputs target query features. The preset query QueryN can be 600. Additionally, the input is a target feature vector fused from joint prompt features, first visual features, and second visual features. M can immediately indicate the number of tasks to be performed. During the training phase, M can be 80, representing 80 business categories. During the inference phase, M is user-defined. The module can internally employ a 4-head cross-attention structure. The similarity matrix between the target and the prompt is calculated. The calculation formula is: After layer normalization and linear projection, the similarity features can output the final matching score. Combine the bounding box features output by the regression branch The Hungarian matching algorithm is used to match and align the predicted bounding boxes with the ground truth bounding boxes in a bipartite graph. The FocalLoss loss function can be chosen to mitigate the imbalance between foreground and background samples. The classification loss calculation formula is shown below: Among them, fixed hyperparameters .

[0076] In at least one embodiment, during the zero-shot inference phase, the user can input entirely new category text description information, and the model can encode and generate entirely new prompt features in real time. The target matching score can be calculated without updating the network model parameters throughout the process, achieving zero-sample target detection without training. It can abandon the fixed category probability output mode and dynamically retrieve and match the corresponding target based on the prompt features.

[0077] This embodiment abandons the reliance of traditional closed-set detectors on fixed category classification heads, and instead uses the target query features and dynamically generated target feature vectors to calculate similarity, enabling the model to have true zero-shot generalization ability and to identify new category targets that have not appeared in the training set in real time.

[0078] The method for determining the object in this application will be explained below with reference to specific embodiments.

[0079] Figure 3 This is a flowchart illustrating the method for determining the object in this optional example, such as... Figure 3 As shown, the process of determining this object may include the following steps:

[0080] Step S302, Data collection and annotation;

[0081] The system acquires low-altitude aerial image data via drones, low-altitude cameras, or airborne vision devices. Users annotate targets with bounding boxes through an online annotation system and input corresponding text descriptions, including target name, attribute features, and scene semantics. The system also supports uploading sample target images as visual prompts.

[0082] Step S304: Construct a training image-text joint prompt dataset;

[0083] The system can organize the original image, target bounding box annotations, text prompts, and example image prompts into a unified low-altitude dedicated image-text joint cue dataset. The text prompts describe the semantic information of the targets, while the image prompts provide prior information about the target appearance. The dataset format is: {Original Image I, Target Bounding Box Set} Text prompt collection Visual cue set }, where each target box Associate the corresponding text prompt and / or visual cues v k (Specific categories of low-altitude vehicles: such as vehicles, pedestrians, boats, animals, etc.);

[0084] Step S306, combined image and text prompt training;

[0085] The system inputs both text and image prompts into the image-text joint prompt encoding module to generate joint prompt features, which are then jointly trained with the RT-DETR detection network. A multi-scale object supervision strategy is employed during training to enhance the detection capability of small objects at long distances. Training is performed on training units, supporting mixed-precision training and gradient accumulation.

[0086] Step S308, Model Lightweighting and Deployment;

[0087] After training, the model .pth file with the highest performance on the test set is selected, and then quantization (FP16), structured pruning (channel importance ranking based on L1 norm, pruning ratio of 30%) and format conversion operations (ONNX→Ascend) are performed to generate the inference model supported by the inference unit and automatically deploy it to the inference unit device.

[0088] Step S310: Integrate drone video data for real-time inference;

[0089] The inference unit receives images from a camera or drone in real time and sequentially performs image preprocessing (distortion correction, deblurring, normalization), image-text cue encoding, open-set RT-DETR inference, and result output. The inference task has the highest scheduling priority to ensure real-time performance.

[0090] Step S312, incremental learning and model update.

[0091] When the system receives new target categories or new scene data, it automatically starts the EWC incremental learning process to update non-critical parameters while preserving critical parameters from previous tasks. After training is complete, it automatically performs model conversion and hot-update deployment without requiring system downtime.

[0092] In at least one embodiment, a heterogeneous integrated hardware architecture for training and inference units is adopted. A single-machine closed-loop process—from data annotation, online training, model conversion, low-latency inference, to incremental updates—is achieved via a high-speed internal bus. The core is the expansion of the real-time closed-set RT-DETR into an open-set image-text joint prompt detection model, complemented by key modules such as low-altitude image enhancement, small target reinforcement, EWC anti-forgetting incremental learning, and training-inference collaborative scheduling, along with a series of functions including domestic adaptation. In this embodiment, a schematic diagram of the object determination system can be found in [reference needed]. Figure 4 , Figure 4 This is a schematic diagram of the structure of the object determination system according to an embodiment of this application, such as... Figure 4 As shown, it can include a data acquisition and online annotation module, an online training module for training units, a graph-text joint prompting encoding module, an open set RT-DETR detection module, a lightweight model and deployment module, a real-time inference module for inference units, an EWC incremental learning module, and a training-inference integrated collaborative scheduling module.

[0093] In at least one embodiment, the scheduling strategy can employ a priority-preemptive time-slice round-robin mechanism: inference tasks are assigned a fixed priority P0 (highest) and a time slice of 50ms. Inference requests that fail to complete within the time slice are placed into a high-priority queue. Training resources are immediately preempted upon arrival of an inference task to ensure the continuity of real-time detection services.

[0094] The training task is set to priority P1, scheduled only when the inference queue is empty, with a single training iteration time slice of 200ms. A checkpoint is forcibly saved and resources released upon completion. The training task employs a breakpoint resumption mechanism, resuming from the most recent checkpoint after being preempted. The model conversion task (quantization / distillation / format conversion) is set to priority P2, executing only in the background during system idle periods (default 2:00 AM - 6:00 AM), without consuming resources during business hours. Resource isolation is set to the lowest priority, with training and inference units physically isolated. Data is exchanged via a shared memory circular buffer (256MB, double-buffered design) to avoid PCIe bus contention. An atomic replacement mechanism is used during model updates; the inference unit switches to the new model after completing the current batch of inference upon receiving an update signal, with a switching latency far less than 5ms. Task isolation scheduling is achieved through a training-inference collaborative scheduling engine, with inference tasks executed first, and training and inference not preempting each other's resources. Compared to existing scheduling strategies that do not distinguish task priorities, this embodiment ensures stable inference latency, meeting the continuity requirements of low-altitude surveillance operations.

[0095] Through the above embodiments, by using lightweight model distillation (pruning 30%-50% + INT8 quantization), low-altitude image preprocessing optimization (operator fusion), and hardware acceleration of the inference unit, the end-to-end inference latency of a single 512×512 image can be controlled within a low latency range. Compared to the latency of approximately 150ms / image on a T4 GPU in existing technologies, this embodiment, with its deep optimization of the edge NPU, can meet the real-time detection requirements of low-altitude environments; it also performs image enhancement and model adaptation for aerial distortion, motion blur, and small targets. Compared to existing technologies, this embodiment, by introducing low-altitude-specific structures such as DWT frequency domain enhancement and a small target enhancement module, can significantly improve the robustness of recognition in degraded scenarios.

[0096] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0098] According to another aspect of the embodiments of this application, an object determination apparatus is also provided. This object determination apparatus can be used to implement the object determination method provided in the above embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0099] Figure 5 This is a structural block diagram of an optional object determining device according to an embodiment of this application, such as... Figure 5As shown, the device for determining the object includes:

[0100] The receiving module 502 is used to receive text information and a reference image input by the target user, wherein the text information is used to represent text description information of the target object to be determined, and the reference image is used to provide appearance information of the target object;

[0101] The first determining module 504 is used to determine fusion features based on the text information and the reference image;

[0102] The second determining module 506 is used to determine a first visual feature based on the reference image, wherein the first visual feature is a feature obtained by enhancing the appearance information.

[0103] The third determining module 508 is used to determine a second visual feature based on the reference image and the fusion feature, wherein the second visual feature is a feature obtained by enhancing the image of a first object included in the reference image, and the number of pixels of the first object is less than or equal to a preset number.

[0104] The fourth determining module 510 is used to determine the target feature vector based on the first visual feature, the second visual feature, and the fused feature;

[0105] The fifth determining module 512 is used to determine the target object from the real-time image based on the target feature vector.

[0106] In an exemplary embodiment, the second determining module 506 may determine the first visual feature based on the reference image by performing a multi-scale feature extraction operation on the reference image to obtain multiple hierarchical features; determining a first hierarchical feature included in the multiple hierarchical features, wherein the first hierarchical feature is used to describe the visual features of the reference image; and determining the first visual feature based on the first hierarchical feature.

[0107] In an exemplary embodiment, the second determining module 506 may determine the first visual feature based on the first hierarchical feature in the following manner: extracting the frequency domain feature of the first hierarchical feature; reducing the dimensionality of the frequency domain feature to obtain a first feature; generating a first weight based on the first feature; fusing the frequency domain feature with the first weight to obtain a second feature; recombining the second feature to obtain a third feature; and fusing the first hierarchical feature with the third feature to obtain the first visual feature.

[0108] In an exemplary embodiment, the third determining module 508 may determine the second visual feature based on the reference image and the fused features in the following manner: performing multi-scale feature extraction on the reference image to obtain multiple hierarchical features; determining a first hierarchical feature, a second hierarchical feature, and a third hierarchical feature included in the multiple hierarchical features, wherein the first hierarchical feature and the second hierarchical feature are used to describe the visual features of the reference image, and the third hierarchical feature is used to extract semantic information in the reference image; determining a first sub-visual feature based on the first hierarchical feature, the third hierarchical feature, and the fused features; determining a second sub-visual feature based on the second hierarchical feature, the third hierarchical feature, and the fused features; and determining the second visual feature based on the first sub-visual feature and the second sub-visual feature.

[0109] In an exemplary embodiment, the third determining module 508 can determine the first sub-visual feature based on the first-level feature, the third-level feature, and the fused feature in the following manner: extracting the context features of the first-level feature to obtain a fourth feature; fusing the fourth feature with the third-level feature to obtain a fifth feature; determining a second weight based on the fused feature; fusing the fifth feature with the second weight to obtain a sixth feature; performing feature extraction on the sixth feature to obtain a seventh feature; extracting the gradient features of the sixth feature to obtain an eighth feature; and concatenating the eighth feature, the seventh feature, and the first-level feature to obtain the first sub-visual feature.

[0110] In an exemplary embodiment, the first determining module 504 may determine the fusion feature based on the text information and the reference image in the following manner: determining text features based on the text information; determining image features based on the reference image; and determining the fusion feature based on the text features and the image features.

[0111] In an exemplary embodiment, the first determining module 504 can determine the fusion feature based on the text features and the image features in the following manner: performing a channel concatenation operation on the text features and the image features to obtain a concatenated feature; determining a third weight corresponding to the text features and a fourth weight corresponding to the image features; performing a mapping operation on the image features to obtain a ninth feature, wherein the mapping operation is used to align the first semantic space of the image features with the second semantic space of the text features; weighting the third weight with the text features to obtain a first sub-fusion feature; weighting the fourth weight with the ninth feature to obtain a second sub-fusion feature; and determining the sum of the first sub-fusion feature and the second sub-fusion feature as the fusion feature.

[0112] In an exemplary embodiment, the fifth determining module 512 can determine the target object from the real-time image based on the target feature vector by: determining the target query feature of the real-time image; determining the similarity score between the target query feature and the target feature vector; and determining the target object from the real-time image based on the similarity score.

[0113] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0114] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the steps in any of the above method embodiments when it is run.

[0115] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0116] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to perform the steps of any of the method embodiments described above via the computer program. In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0117] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0118] According to another aspect of the embodiments of this application, a computer program product is also provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit 601, it performs various functions provided in the embodiments of this application. The sequence numbers of the embodiments of this application above are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0119] Figure 6A schematic block diagram of a computer system architecture for implementing embodiments of the present application is shown. Figure 6 As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which performs various appropriate actions and processes based on programs stored in ROM 602 or loaded into RAM 603 from storage section 608. Random Access Memory 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0120] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card, such as a local area network card or modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0121] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit 601, it performs various functions defined in the system of this application.

[0122] It should be noted that, Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0123] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0124] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining an object, characterized in that, include: The system receives text information input by a target user and a reference image, wherein the text information is used to represent textual description information of the target object to be determined, and the reference image is used to provide appearance information of the target object; Based on the text information and the reference image, fusion features are determined; A first visual feature is determined based on the reference image, wherein the first visual feature is a feature obtained by enhancing the appearance information; A second visual feature is determined based on the reference image and the fusion features, wherein the second visual feature is a feature obtained by enhancing the image of a first object included in the reference image, and the number of pixels of the first object is less than or equal to a preset number. The target feature vector is determined based on the first visual feature, the second visual feature, and the fused feature. The target object is determined from the real-time image based on the target feature vector.

2. The method according to claim 1, characterized in that, Determining the first visual feature based on the reference image includes: Multi-scale feature extraction is performed on the reference image to obtain multiple levels of features; A first-level feature is determined among a plurality of said hierarchical features, the first-level feature being used to describe the visual features of the reference image; The first visual feature is determined based on the first level feature.

3. The method according to claim 2, characterized in that, Determining the first visual feature based on the first-level feature includes: Extract the frequency domain features of the first-level features; The frequency domain features are reduced in dimensionality to obtain the first feature; A first weight is generated based on the first feature; The frequency domain features are fused with the first weight to obtain the second feature; The second feature is recombined to obtain the third feature; The first-level feature is fused with the third feature to obtain the first visual feature.

4. The method according to claim 1, characterized in that, Determining the second visual feature based on the reference image and the fused features includes: Multi-scale feature extraction is performed on the reference image to obtain multiple levels of features; A first-level feature, a second-level feature, and a third-level feature are determined among multiple hierarchical features. The first-level feature and the second-level feature are used to describe the visual features of the reference image, and the third-level feature is used to extract semantic information from the reference image. The first sub-visual feature is determined based on the first-level feature, the third-level feature, and the fused feature; The second sub-visual feature is determined based on the second-level feature, the third-level feature, and the fusion feature; The second visual feature is determined based on the first sub-visual feature and the second sub-visual feature.

5. The method according to claim 4, characterized in that, The first sub-visual feature is determined based on the first-level feature, the third-level feature, and the fused feature, including: Extract the context features of the first-level features to obtain the fourth feature; The fourth feature is fused with the third-level feature to obtain the fifth feature; The second weight is determined based on the fusion features; The fifth feature is fused with the second weight to obtain the sixth feature; The sixth feature is subjected to feature extraction to obtain the seventh feature; Extract the gradient features of the sixth feature to obtain the eighth feature; The eighth feature, the seventh feature, and the first-level feature are concatenated to obtain the first sub-visual feature.

6. The method according to claim 1, characterized in that, Determining fusion features based on the text information and the reference image includes: Determine text features based on the text information; Image features are determined based on the reference image; The fusion feature is determined based on the text features and the image features.

7. The method according to claim 6, characterized in that, Determining the fusion feature based on the text features and the image features includes: The text features and the image features are concatenated by channels to obtain the concatenated features; Determine the third weight corresponding to the text feature and the fourth weight corresponding to the image feature; A mapping operation is performed on the image features to obtain a ninth feature. The mapping operation is used to align the first semantic space of the image features with the second semantic space of the text features. The third weight is weighted together with the text feature to obtain the first sub-fusion feature; The fourth weight is weighted together with the ninth feature to obtain the second sub-fusion feature; The sum of the first sub-fusion feature and the second sub-fusion feature is determined as the fusion feature.

8. The method according to claim 1, characterized in that, Determining the target object from the real-time image based on the target feature vector includes: Determine the target query features of the real-time image; Determine the similarity score between the target query feature and the target feature vector; The target object is determined from the real-time image based on the similarity score.

9. An object determining device, characterized in that, include: A receiving module is used to receive text information and a reference image input by a target user, wherein the text information is used to represent textual description information of a target object to be determined, and the reference image is used to provide appearance information of the target object; The first determining module is used to determine fusion features based on the text information and the reference image; The second determining module is used to determine a first visual feature based on the reference image, wherein the first visual feature is a feature obtained by enhancing the appearance information; The third determining module is used to determine a second visual feature based on the reference image and the fusion feature, wherein the second visual feature is a feature obtained by enhancing the image of the first object included in the reference image, and the number of pixels of the first object is less than or equal to a preset number. The fourth determining module is used to determine the target feature vector based on the first visual feature, the second visual feature, and the fused feature; The fifth determining module is used to determine the target object from the real-time image based on the target feature vector.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images