Target detection method and device, equipment, storage medium and program product

Abstract

The invention discloses a target detection method and device, equipment, a storage medium and a program product. The method is applied to the intelligent wearable device and comprises the steps that voice information output by a user and a first image shot by the intelligent wearable device are acquired, and the voice information comprises information of a target object; target areas corresponding to the entity objects are determined in the first image; cutting the first image according to the plurality of target areas to obtain a plurality of second images; extracting visual features corresponding to the plurality of second images to obtain a plurality of first visual features, and extracting a first text feature of the target object from the voice information; and detecting a target object based on a second image corresponding to a target visual feature meeting a preset similar condition with the first text feature in the plurality of first visual features. According to the target detection method provided by the embodiment of the invention, the specific target object can be detected from the whole image according to the intention of the user, and the use experience of the user on the intelligent wearable equipment is improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a target detection method, apparatus, device, storage medium and program product. Background Technology

[0002] With the continuous advancement of optical display technology and artificial intelligence technology, new payment wallet models such as wearable devices (e.g., smart glasses) have emerged. Compared to devices such as mobile phones and cameras, smart glasses equipped with cameras have a more natural and focused human eye perspective for shooting, and their applications are becoming increasingly widespread.

[0003] Currently, smart glasses equipped with neural network models (such as the Llama model) can recognize images captured by the smart glasses, but they can only recognize the entire image and cannot detect specific target objects from the entire image based on user intent. Therefore, there is an urgent need for a target detection method for smart wearable devices that can detect specific target objects from the entire image based on user intent, thereby improving the user experience of smart wearable devices. Summary of the Invention

[0004] This application provides a target detection method, apparatus, device, computer-readable storage medium, and computer program product, which can detect specific target objects from the entire image according to user intent, thereby improving the user experience of smart wearable devices.

[0005] In a first aspect, embodiments of this application provide a target detection method applied to a smart wearable device, the method comprising:

[0006] Acquire user-output voice information and a first image captured by the smart wearable device, wherein the voice information includes information about the target object;

[0007] In the first image, the target regions corresponding to each of the multiple entity objects are determined;

[0008] The first image is cropped according to multiple target regions to obtain multiple second images;

[0009] Visual features corresponding to the multiple second images are extracted to obtain multiple first visual features, and text features of the target object are extracted from the speech information to obtain first text features;

[0010] The target object is detected based on the second image corresponding to the target visual feature that satisfies the preset similarity condition of the first text feature among the plurality of first visual features.

[0011] In one possible implementation, the smart wearable device deploys an object detection model, which includes a feature extraction layer and a region proposal network layer. The step of determining the target regions corresponding to multiple entity objects in the first image includes:

[0012] The feature extraction layer is used to extract the channel information of each pixel in the first image to obtain a first feature map.

[0013] Using the region proposal network layer, multiple candidate regions are generated based on the first feature map, and the probability that the candidate regions include entity objects is predicted.

[0014] The candidate regions corresponding to the probabilities greater than a preset probability threshold among the multiple probabilities are determined as the target regions.

[0015] In one possible implementation, the object detection model further includes a pixel coordinate encoding layer. Before generating multiple candidate regions based on the first feature map using the region proposal network layer and predicting the probability that the candidate regions contain entity objects, the method further includes:

[0016] Obtain the coordinate information of each pixel in the first image;

[0017] Using the pixel coordinate encoding layer, for each pixel, the coordinate information is added to the channel information of the first feature map to obtain the second feature map;

[0018] The step of using the region proposal network layer to generate multiple candidate regions based on the first feature map and predicting the probability that the candidate regions contain entity objects includes:

[0019] Using the region proposal network layer, multiple candidate regions are generated based on the second feature map, and the probability that the candidate regions include entity objects is predicted.

[0020] In one possible implementation, detecting the target object based on the second image corresponding to the target visual feature among the plurality of first visual features that satisfies a preset similarity condition to the first text feature includes:

[0021] The first text feature and multiple first visual features are mapped to the same vector space to obtain the second text feature and multiple second visual features;

[0022] The target object is detected based on the second image corresponding to the target visual feature that satisfies the preset similarity condition of the second text feature among the plurality of second visual features.

[0023] In one possible implementation, before detecting the target object based on the second image corresponding to the target visual feature that satisfies a preset similarity condition with the first text feature among the plurality of first visual features, the method further includes:

[0024] A third image is obtained by capturing the user's gaze point area;

[0025] Extract the visual features of the third image to obtain the third visual features;

[0026] The method of detecting the target object based on the second image corresponding to the target visual feature that satisfies a preset similarity condition with the first text feature among the plurality of first visual features includes:

[0027] The target object is detected based on the second image corresponding to the target visual feature among the plurality of first visual features and the third visual features that satisfies the preset similarity condition with the first text features.

[0028] In one possible implementation, before extracting the text features of the target object from the speech information to obtain the first text features, the method further includes:

[0029] The voice information is automatically recognized to obtain text information;

[0030] The text information is semantically parsed to determine the target object;

[0031] The step of extracting text features of the target object from the speech information to obtain the first text feature includes:

[0032] The text features of the target object are extracted from the text information to obtain the first text feature.

[0033] In one possible implementation, the speech information includes information about a target pronoun referring to the target object. Before semantically parsing the text information to determine the target object, the method further includes:

[0034] Acquire dialogue information between the user and the smart wearable device within a preset time period;

[0035] Extract entity objects from the dialogue information according to the order in which they appear.

[0036] The step of semantically parsing the text information to determine the target object includes:

[0037] The text information is semantically parsed to determine the target pronoun;

[0038] The newly extracted entity object is identified as the target object corresponding to the target pronoun.

[0039] In one possible implementation, the smart wearable device deploys an object detection model, which further includes a visual encoder and a text encoder. The step of extracting visual features corresponding to the plurality of second images to obtain plurality of first visual features, and extracting text features of the target object from the speech information to obtain first text features, includes:

[0040] The visual encoder is used to extract the visual features corresponding to the plurality of second images respectively to obtain the plurality of first visual features;

[0041] Using the text encoder, the text features of the target object are extracted from the text information to obtain the first text feature.

[0042] In one possible implementation, detecting the target object based on the second image corresponding to the target visual feature among the plurality of first visual features that satisfies a preset similarity condition to the first text feature includes:

[0043] When there are multiple target visual features, the scene type of the first image and the object category of the entity objects in the second image corresponding to the multiple target visual features are obtained;

[0044] The target object is detected based on the second image corresponding to the object category that matches the scene type.

[0045] In one possible implementation, the method further includes:

[0046] Obtain multiple first image samples segmented from the original image samples and a first text description sample corresponding to each first image sample;

[0047] The visual features of the first image sample are extracted using the initial visual encoder in the initial object detection model to obtain the fourth visual features;

[0048] The text features of the first text description sample are extracted using the initial text encoder in the initial target detection model to obtain the third text features;

[0049] By comparing loss functions, the value of the first loss function is determined based on the similarity between the third text feature and the fourth visual feature. The comparison loss function is used to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs.

[0050] If the first loss function value is greater than or equal to the first preset threshold, the model parameters of the initial object detection model are adjusted, and the process of extracting visual features of the first image sample using the initial visual encoder in the initial object detection model is repeated until the first loss function value is less than the first preset threshold, and the initial object detection model is determined as the object detection model.

[0051] In one possible implementation, the initial object detection model further includes an initial feature extraction layer and an initial pixel coordinate encoding layer. The step of extracting visual features from the first image sample using the initial visual encoder in the initial object detection model to obtain a fourth visual feature includes:

[0052] The channel information of each pixel in the first image sample is extracted using the initial feature extraction layer to obtain the first feature map sample;

[0053] Obtain the coordinate information of each pixel in the first image sample;

[0054] Using the initial pixel coordinate encoding layer, for each pixel, the coordinate information sample is added to the channel information of the first feature map sample to obtain the second feature map sample;

[0055] The visual features of the second feature map sample are extracted using the initial visual encoder to obtain the fourth visual feature.

[0056] In one possible implementation, after obtaining the target detection model, the method further includes:

[0057] Obtain multiple second image samples segmented from the original image samples and a second text description sample corresponding to each second image sample;

[0058] The first visual encoder and the first text encoder in the object detection model are used as teacher models;

[0059] Obtain a student model corresponding to the teacher model. The student model includes a second visual encoder corresponding to the first visual encoder and a second text encoder corresponding to the first text encoder. The parameter value of the target parameter of the second visual encoder is less than the parameter value of the target parameter of the first visual encoder, and the parameter value of the target parameter of the second text encoder is less than the parameter value of the target parameter of the first text encoder.

[0060] The visual features of the second image sample are extracted using the first visual encoder to obtain the fifth visual feature;

[0061] The visual features of the second image sample are extracted using the second visual encoder to obtain the sixth visual feature;

[0062] The first text encoder is used to extract the text features of the second text description sample to obtain the fourth text features;

[0063] The text features of the second text description sample are extracted using the second text encoder to obtain the fifth text feature;

[0064] The second loss function value is determined based on the similarity between the sixth visual feature and the fifth text feature, the similarity between the sixth visual feature and the fifth visual feature, and the similarity between the fifth text feature and the fourth text feature;

[0065] If the value of the second loss function is greater than or equal to the second preset threshold, the model parameters of the student model are adjusted, and the process of extracting the visual features of the second image sample using the first visual encoder is repeated until the value of the second loss function is less than the second preset threshold, at which point the student model is determined as the target detection model.

[0066] In one possible implementation, after detecting the target object, the method further includes:

[0067] Retrieve a list of product links corresponding to the target object;

[0068] Based on the product link list, display the target product link list;

[0069] Receive the user's first input to select a target product link from the list of target product links;

[0070] In response to the first input, a target order corresponding to the target product link is created;

[0071] Receive a second input from the user for payment of the target order;

[0072] In response to the second input, the target order is paid.

[0073] In one possible implementation, obtaining the list of product links corresponding to the target object includes:

[0074] Call the e-commerce platform's search interface to search for a list of product links corresponding to the target object.

[0075] In one possible implementation, displaying the target product link list based on the product link list includes:

[0076] Obtain the user's purchase preference information and the product information corresponding to multiple products in the product link list;

[0077] Based on the purchase preference information and the product information, select multiple target products that the user is interested in from the plurality of products;

[0078] Based on the multiple target products, a list of target product links is determined;

[0079] Display the list of links to the target product.

[0080] Secondly, embodiments of this application provide a target detection device applied to a smart wearable device, the device comprising:

[0081] The first acquisition module is used to acquire voice information output by the user and a first image captured by the smart wearable device, wherein the voice information includes information about the target object;

[0082] The first determining module is used to determine the target regions corresponding to each of the multiple entity objects in the first image;

[0083] A cropping module is used to crop the first image according to multiple target regions to obtain multiple second images;

[0084] The first extraction module is used to extract visual features corresponding to the plurality of second images respectively to obtain a plurality of first visual features, and to extract text features of the target object from the speech information to obtain first text features;

[0085] The detection module is used to detect the target object based on the second image corresponding to the target visual feature that satisfies a preset similarity condition with the first text feature among the plurality of first visual features.

[0086] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions;

[0087] When the processor executes the computer program instructions, it implements any of the possible implementations of the first aspect described above.

[0088] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the method in any of the possible implementations of the first aspect described above.

[0089] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a method as described in any of the possible implementations of the first aspect above.

[0090] The target detection method, apparatus, device, computer-readable storage medium, and computer program product in this application embodiment, by determining target regions corresponding to multiple entity objects in a first image captured by a smart wearable device, and cropping the first image according to the multiple target regions to obtain multiple second images, can segment a complete first image into multiple second images that may include target objects. By extracting visual features corresponding to the multiple second images respectively to obtain multiple first visual features, and extracting text features of the target object from the voice information to obtain first text features, and by detecting the target object based on the second image corresponding to the target visual features that satisfy preset similarity conditions with the first text features among the multiple first visual features, a specific target object can be detected from the entire image according to the user's intent, thereby improving the user experience of the smart wearable device. Attached Figure Description

[0091] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0092] Figure 1 This is a schematic flowchart of a target detection method provided in an embodiment of this application;

[0093] Figure 2 This is a schematic diagram of a target detection method provided in an embodiment of this application;

[0094] Figure 3 This is a flowchart illustrating another target detection method provided in an embodiment of this application;

[0095] Figure 4 This is a schematic diagram of another target detection method provided in an embodiment of this application;

[0096] Figure 5 This is a schematic diagram illustrating a process for purchasing the same product offline using a smart wearable device, as provided in an embodiment of this application.

[0097] Figure 6 This is a schematic diagram of the structure of a target detection device provided in an embodiment of this application;

[0098] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0099] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0100] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0101] Furthermore, the acquisition, storage, use, and processing of data in this application's technical solution all comply with relevant national laws and regulations.

[0102] With the continuous advancement of optical display technology and artificial intelligence technology, new payment wallet models such as wearable devices (e.g., smart glasses) have emerged. Compared to devices such as mobile phones and cameras, smart glasses equipped with cameras have a more natural and focused human eye perspective for shooting, and their applications are becoming increasingly widespread.

[0103] However, smart wearable devices are not yet integrated with online e-commerce. Integrating smart wearable devices with online e-commerce would provide users with a more convenient and efficient payment experience.

[0104] Based on this, the inventive concept of this application is as follows: Taking a visual e-commerce scenario as an example, when a user sees an item they like offline and wants to buy it online, it is hoped that the smart wearable device can call the camera of the smart wearable device to capture the image within the user's current field of view after receiving the user's instruction (such as "I want to buy this water cup"), detect the object that the user wants to buy from the image, and transmit the image information of the object to the open interface of the e-commerce platform to obtain a list of similar products for the user to select and place an order.

[0105] Currently, smart glasses equipped with neural network models (such as the Llama model) can recognize images captured by the smart glasses, but they can only recognize the entire image and cannot detect specific target objects from the entire image based on user intent. In fact, there may be multiple objects in a video stream from the human eye's perspective. Therefore, accurately identifying and segmenting the object referred to by the user under various morphological commands is the core technical challenge.

[0106] Therefore, there is an urgent need for a target detection method for smart wearable devices to detect specific target objects from the entire image according to the user's intent, thereby realizing the integration of smart wearable devices with online e-commerce and improving the user experience of smart wearable devices.

[0107] In related technologies, e-commerce systems can first automatically segment different items in a captured image and generate names of similar items. Then, users manually select items or adjust the segmented image to finally confirm the target item. However, this method relies solely on image data, making it highly dependent on the shooting angle, lighting conditions, and image clarity. It can only recommend similar products based on image similarity, and users cannot provide further details about the items through voice or other means. This results in low accuracy in product recommendations and a poor user experience for existing e-commerce systems.

[0108] In addition, some target detection methods in related technologies have large model sizes, which cannot be directly deployed on the hardware of smart wearable devices due to the size and power consumption limitations of smart wearable devices.

[0109] To address the aforementioned challenges, this application proposes a multimodal fusion referential target detection method for smart wearable devices. Based on the camera and voice sensor mounted on the smart wearable device, a multimodal fusion semantic search model is built with inputs of multimodal information such as natural language and images, achieving accurate referential image matching and segmentation.

[0110] Based on this, and to address the problems of the prior art, embodiments of this application provide a target detection method, apparatus, device, computer-readable storage medium, and computer program product. The target detection method can be applied to scenarios combining smart wearable devices with online e-commerce.

[0111] The target detection method provided in the embodiments of this application will be introduced below.

[0112] Figure 1 A schematic flowchart of a target detection method provided in an embodiment of this application is shown. This target detection method can be executed by a smart wearable device. Figure 1 As shown, the target detection method provided in this application includes the following steps:

[0113] S110. Acquire the voice information output by the user and the first image captured by the smart wearable device, wherein the voice information includes information about the target object;

[0114] S120. Determine the target regions corresponding to each of the multiple entity objects in the first image;

[0115] S130. Cropping the first image according to multiple target regions yields multiple second images;

[0116] S140. Extract the visual features corresponding to multiple second images to obtain multiple first visual features, and extract the text features of the target object from the speech information to obtain the first text features;

[0117] S150. Detect the target object based on the second image corresponding to the target visual feature that satisfies the preset similarity condition of the first text feature among multiple first visual features.

[0118] The target detection method in this embodiment of the application determines target regions corresponding to multiple entity objects in a first image captured by a smart wearable device, and crops the first image according to the multiple target regions to obtain multiple second images. This can segment the complete first image into multiple second images that may include target objects. By extracting visual features corresponding to the multiple second images respectively, multiple first visual features are obtained, and text features of the target object in the voice information are extracted to obtain first text features. Based on the second image corresponding to the target visual features that meet the preset similarity conditions of the first text features among the multiple first visual features, the target object is detected. This enables the detection of specific target objects from the entire image according to user intent, thereby improving the user experience of the smart wearable device.

[0119] The specific implementation methods for each of the above steps are described below.

[0120] In some embodiments, in S110, the first image can be an image captured by the smart wearable device within the user's field of view. The first image may include multiple entity objects. Entity objects can be non-background objects. The target object can be any one of the multiple entity objects. The voice information output by the user can be used to instruct the smart wearable device to identify and detect the target object from the first image. The target object information included in the voice information may include at least one of the target object's name and referential information. The referential information of the target object may include pronouns, directional location information, color information, etc. The voice information may be, for example, "I want to buy this water cup," "I want to buy the water cup in the upper left corner," "I want to buy the same item in the upper left corner," "I want to buy this," etc.

[0121] As an example, upon receiving a user's instruction to take an image, the smart wearable device can capture a first image within the user's field of vision. This instruction can be a touch command, a behavioral command, or a voice command, without limitation. A touch command can be issued by the user touching a "photo-taking control," which can be a physical or virtual control. A behavioral command can be the user gazing at a certain location for a preset duration. Furthermore, a voice command may or may not include information about the target object. That is, a voice command can simply instruct the smart wearable device to take a picture, such as "take a picture," or it can instruct the smart wearable device to first capture a first image and then detect the target object from the first image, such as "I want to buy this water cup." If the voice command is only used to instruct the smart wearable device to take a picture, then after the smart wearable device captures the first image, it can receive voice information from the user that includes the target object.

[0122] In some embodiments, in S120, the method of determining the target regions corresponding to each of the multiple entity objects in the first image may include: first identifying multiple entity objects from the first image, and then determining the regions where the multiple entity objects are located as multiple target regions; and first determining multiple candidate regions in the first image, and then determining the regions of the multiple candidate regions that include entity objects as target regions.

[0123] In some embodiments, a target detection model may be deployed in the smart wearable device, which can determine the target regions corresponding to multiple entity objects in the first image.

[0124] Due to the hardware limitations of smart wearable devices and the need for fast response times in user interaction, excessive computational resources and long inference times required by object detection models can significantly impact the user experience. Therefore, to improve the efficiency of target region determination and adapt to smart wearable devices, in some embodiments, the object detection model includes a feature extraction layer and a region proposal network layer. Based on this, the aforementioned S120 may specifically include:

[0125] The first feature map is obtained by extracting the channel information of each pixel in the first image using a feature extraction layer.

[0126] Using a region proposal network layer, multiple candidate regions are generated based on the first feature map, and the probability that the candidate regions include entity objects is predicted.

[0127] The candidate region corresponding to the probability greater than a preset probability threshold among multiple probabilities is determined as the target region.

[0128] Here, the feature extraction layer can be a convolutional neural network, which can extract the channel information of each pixel in the first image to obtain the first feature map I∈R. H*W*3 Here, H represents the height of the first feature map, i.e., the number of rows of pixels; W represents the width of the first feature map, i.e., the number of columns of pixels; and 3 represents the three channel information of each pixel in the first feature map. These three channel information can include the pixel value of the red channel (R value), the pixel value of the green channel (G value), and the pixel value of the blue channel (B value).

[0129] Region Proposal Network (RPN) layers can be used to efficiently generate candidate regions with a small number of parameters, offering advantages such as lightweight design and low computational cost. The RPN layer can extract multiple candidate regions from the first feature map, each corresponding to an image dataset {I1, I2…I...}. n}, each image data I i The region (i = 1…n) can include the location information of the candidate region in the first image and the probability that the candidate region contains an entity object. i This can be denoted as {w1, h1, w2, h2, p}. Here, (w1, h1) represents the coordinates of the top-left corner of the candidate region, (w2, h2) represents the coordinates of the bottom-right corner of the candidate region, and p represents the probability that the candidate region contains an entity object. The higher the probability, the greater the likelihood that the candidate region contains an entity object. Therefore, by determining multiple candidate regions and their corresponding probabilities, the candidate regions with probabilities greater than a preset probability threshold can be identified as target regions, thus obtaining the target regions corresponding to multiple entity objects.

[0130] Since the region proposal network has fewer parameters, it has the advantages of being lightweight and having low computational cost. Therefore, the embodiments of this application can improve the efficiency of determining the target region by using a region proposal network with fewer parameters, and can be adapted to smart wearable devices.

[0131] Furthermore, in practice, a user's voice information may include the directional location information of the target object, such as "I want to buy the water cup in the upper left corner," or "I want to buy the same item in the upper left corner." Therefore, to improve the generalization ability and accuracy of subsequent object detection, in some embodiments, the object detection model may also include a pixel coordinate encoding layer. Based on this, before generating multiple candidate regions based on the first feature map using the region proposal network layer described above, the method may further include:

[0132] Obtain the coordinate information of each pixel in the first image;

[0133] By using a pixel coordinate encoding layer, for each pixel, the coordinate information is added to the channel information of the first feature map to obtain the second feature map.

[0134] Based on this, the aforementioned region proposal network layer generates multiple candidate regions based on the first feature map and predicts the probability that the candidate regions contain entity objects. Specifically, this can include:

[0135] By utilizing a region proposal network layer, multiple candidate regions are generated based on the second feature map, and the probability of including entity objects in the candidate regions is predicted.

[0136] Here, the first feature map may include the coordinate information of each pixel, but the coordinate information is implicit at this time. For each pixel, the embodiments of this application can extract the implicit coordinate information from the first feature map, and use a pixel coordinate encoding layer to expand the x and y coordinates of the pixel into additional channels, thereby expanding the channel information of the pixel from [R,G,B] to [x,y,R,G,B], that is, converting the implicit coordinates into explicit coordinates, and thus transforming the first feature map I∈R H*W*3 Updated to I∈R H*W*5 .

[0137] This application embodiment explicitly encodes the coordinate information of pixels into channel information to obtain a second feature map, and uses a region proposal network layer to determine multiple target regions. This allows the target regions to possess the location information of entities, so that when the user's speech information contains adverbs indicating direction, target detection can be performed by matching the direction information in the speech information with the location information of entities in the target regions, thereby improving the generalization ability and accuracy of target detection. In other words, explicit coordinate encoding plays a crucial role when user commands are ambiguous.

[0138] Specifically, if the user's voice information includes both directional location information and target object information, such as "I want to buy the water cup in the upper left corner," then matching the directional location information in the voice information with the location information of the entity in the target area can improve the accuracy of object detection. If the user's voice information includes directional location information but not target object information, such as "I want to buy the same item in the upper left corner," then matching the directional location information in the voice information with the location information of the entity in the target area can improve the success rate of object detection, thereby improving the generalization ability of object detection.

[0139] In addition, since both the first feature map and the second feature map include pixel color information, when the user's voice information includes color information, the embodiments of this application can also assist in target detection based on the matching between the color information in the voice information and the color information of the entity object in the target area, thereby distinguishing target objects of different colors and improving the generalization ability and accuracy of target detection.

[0140] In some embodiments, in S130, after determining multiple target regions, the first image can be cropped and segmented according to the multiple target regions to obtain multiple second images. That is, the image data I corresponding to the multiple target regions mentioned above can be used as the image data of the multiple second images. Each second image may include an entity object.

[0141] In some embodiments, in S140, after obtaining multiple second images, visual features corresponding to each of the multiple second images can be extracted from each of the multiple second images to obtain multiple first visual features. Additionally, since the speech information includes information about the target object, text features of the target object can be extracted from the speech information to obtain first text features.

[0142] In some embodiments, the object detection model may further include a visual encoder. Based on this, S140 above may specifically include:

[0143] A visual encoder is used to extract the visual features corresponding to multiple second images to obtain multiple first visual features.

[0144] Here, to adapt to smart wearable devices, the visual encoder can be the backbone network of a lightweight large-scale visual model (TinyVision Transformer, TinyViT). The visual encoder can visually encode the second image to obtain the first visual feature. The first visual feature can be denoted as f. I .

[0145] Based on the above embodiments, the schematic diagram of target detection provided in this application embodiment can be... Figure 2 As shown.

[0146] Furthermore, in order to extract text features of the target object from the speech information, in some embodiments, the method may further include the following steps before S140 described above:

[0147] Automatic speech recognition is performed on the voice information to obtain the text information;

[0148] Perform semantic analysis on the text information to determine the target object.

[0149] Based on this, the above extraction of text features of the target object from speech information to obtain the first text feature can specifically include:

[0150] The text features of the target object are extracted from the text information to obtain the first text feature.

[0151] Here, the smart wearable device may include an Automatic Speech Recognition (ASR) module and a Natural Language Understanding (NLP) module. After receiving the user's voice information, the smart wearable device can first use the ASR module to automatically recognize the voice information to obtain text information, and then use the NLP module to perform semantic analysis on the text information to determine the target object. The text information can be a word sequence text = {t1, t2, ...}, and the target object can be denoted as text. noun .

[0152] Thus, the above-mentioned extraction of text features of the target object from speech information can specifically be the extraction of text features of the target object from text information.

[0153] In some embodiments, the object detection model may further include a text encoder. Based on this, S140 above may specifically include:

[0154] Using a text encoder, text features of the target object are extracted from text information to obtain the first text feature.

[0155] Here, to adapt to smart wearable devices, the text encoder can be the backbone network of a lightweight text transformation model. The text encoder encodes the target object into text, obtaining the first text feature. This first text feature can be denoted as F. textnoun .

[0156] In addition, the speech information may include information about target pronouns used to refer to the target object. These target pronouns may be "this," "that," "it," etc.

[0157] Based on this, in order to improve the generalization ability of object detection, in some embodiments, before performing semantic parsing on the text information and determining the target object as described above, the method may further include:

[0158] Acquire dialogue information between the user and the smart wearable device within a preset time period;

[0159] Extract entity objects from the dialogue information according to the order in which they appear.

[0160] Semantic analysis of text information is performed to determine the target object, including:

[0161] Perform semantic analysis on the text information to determine the target pronoun;

[0162] The newly extracted entity object is identified as the target object corresponding to the target pronoun.

[0163] Here, the natural language understanding module can parse the dialogue information between the user and the smart wearable device in real time, identify entity objects, extract entity objects from the dialogue information according to their order of appearance, and cache entity objects based on their order of appearance. Thus, when semantic parsing of the text information yields a target pronoun referring to the target object, rather than the target object itself, the most recently cached entity object (i.e., the most recently extracted entity object) can be retrieved from the cache and identified as the target object corresponding to the target pronoun.

[0164] This application embodiment pre-caches entity objects mentioned in the dialogue between the user and the smart wearable device within a preset time period. Even when the target pronoun appears in the user's voice information but the target object does not appear, the target object corresponding to the target pronoun can still be determined through caching, thereby improving the generalization ability of target detection.

[0165] In some embodiments, in S150, the preset similarity condition may include a similarity greater than a preset similarity threshold, and a similarity that is at least one of the maximum values ​​among a plurality of similarities.

[0166] As an example, the first text feature can be matched with multiple first visual features to obtain multiple feature pairs. Then, the similarity between each feature pair is determined, and the maximum value among the multiple similarities is determined to satisfy the preset similarity condition. Then, the first visual feature corresponding to the maximum similarity is determined as the target visual feature, the second image corresponding to the target visual feature is determined as the target image, and the entity object in the target image is determined as the target object.

[0167] Since the first text feature is a text feature and the first visual feature is a visual feature, and their feature dimensions are different, in order to improve the accuracy of object detection, in some embodiments, the above S150 may specifically include:

[0168] The first text feature and multiple first visual features are mapped to the same vector space to obtain the second text feature and multiple second visual features;

[0169] The target object is detected based on the second image corresponding to the target visual feature that satisfies the preset similarity condition of the second text feature among multiple second visual features.

[0170] Here, the object detection model can also include a mapping layer. This layer maps the first text features and multiple first visual features to the same vector space, resulting in second text features and multiple second visual features. This achieves multimodal feature fusion, and the similarity between the second text features and the second visual features (after multimodal feature fusion) is then determined. The closer the distance, the greater the similarity; the farther the distance, the smaller the similarity.

[0171] This application embodiment improves the accuracy of similarity by first fusing multimodal features and then determining the similarity between the second text feature and the second visual feature based on the cosine distance between the fused multimodal features. This improves the accuracy of target detection.

[0172] Furthermore, the target visual feature can be one or multiple. If there is only one target visual feature, the entity object in the second image corresponding to that target visual feature can be directly identified as the target object. If there are multiple target visual features, the target object can be further detected based on the scene type of the first image.

[0173] Therefore, in order to improve the accuracy of target detection, in some embodiments, the above-mentioned S150 may specifically include:

[0174] When there are multiple target visual features, obtain the scene type of the first image and the object category of the entity object in the second image corresponding to the multiple target visual features;

[0175] The target object is detected based on the second image corresponding to the object category that matches the scene type.

[0176] Here, the smart wearable device can also deploy scene classification models and image classification models. The scene classification model can identify the scene type of a first image. The input to the scene classification model can be the first image, and the output can be the scene type. The training data for the scene classification model can include image samples and their corresponding scene type labels. Additionally, the image classification model can identify the object category of entities in a second image. The input to the image classification model can be the second image, and the output can be the object category. The training data for the image classification model can include image samples and their corresponding object categories. Both the scene classification model and the image classification model can be lightweight neural network models.

[0177] Therefore, if the scene type is a kitchen scene, kitchen utensils can be prioritized as the target objects. If the scene type is an office scene, office supplies can be prioritized as the target objects.

[0178] The embodiments of this application can improve the accuracy of target detection by detecting target objects based on a second image corresponding to the object category that matches the scene type when there are multiple target visual features.

[0179] Furthermore, to further improve the accuracy of target detection, in some embodiments, the method may further include the following steps before S150 described above:

[0180] The user's gaze point area is captured to obtain a third image;

[0181] Extract the visual features of the third image to obtain the third visual features;

[0182] Based on the second image corresponding to the target visual feature that satisfies a preset similarity condition with the first text feature among multiple first visual features, the target object is detected, including:

[0183] The target object is detected based on the second image corresponding to the target visual feature that meets the preset similarity condition of the first text feature among multiple first visual features and third visual features.

[0184] Here, smart wearable devices can be equipped with eye-tracking modules. These modules can analyze gaze points using sensors such as infrared cameras to determine the user's gaze area. The gaze area is highly likely to include the target object.

[0185] As an example, a smart wearable device can capture the user's gaze area simultaneously with the first image, resulting in a third image. Then, using the feature extraction layer of an object detection model, the channel information of each pixel in the third image can be extracted, yielding a third feature map I. s Using the visual encoder of the object detection model, visual features of the third image can be extracted to obtain the third visual feature. Using the mapping layer of the object detection model, the third visual feature is mapped to the same vector space as the second visual feature and the second text feature to obtain the seventh visual feature. The cosine distance between the seventh visual feature and the second text feature is calculated to obtain the similarity between the seventh visual feature and the second text feature. If the similarity is greater than the similarity between any second visual feature and the second text feature, then the entity object in the third image is identified as the target object.

[0186] This application embodiment can further improve the accuracy of target detection by increasing the user's gaze area when detecting target objects.

[0187] Furthermore, in order to obtain the above-mentioned object detection model, in some embodiments, the object detection method may also include a training process for the object detection model, as detailed below:

[0188] Obtain multiple first image samples segmented from the original image samples and a first text description sample corresponding to each first image sample;

[0189] The visual features of the first image sample are extracted using the initial visual encoder in the initial object detection model to obtain the fourth visual features;

[0190] The text features of the first text description sample are extracted using the initial text encoder in the initial object detection model to obtain the third text features;

[0191] By comparing loss functions, the value of the first loss function is determined based on the similarity between the third text feature and the fourth visual feature. The comparison loss function is used to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs.

[0192] If the first loss function value is greater than or equal to the first preset threshold, adjust the model parameters of the initial object detection model, and return to the initial visual encoder in the initial object detection model to extract the visual features of the first image sample until the first loss function value is less than the first preset threshold, and then determine the initial object detection model as the object detection model.

[0193] Here, the initial object detection model can be the object detection model before training. The initial object detection model and the object detection model can have the same model structure. The initial object detection model can include an initial visual encoder and an initial text encoder. In addition, the contrastive loss function can be as shown in Equation (1):

[0194]

[0195] In equation (1), N can represent the number of the first image samples and their corresponding first text description samples, I i T can represent the i-th fourth visual feature. i I can represent the feature of the i-th third sample. i ·T j I can represent the dot product between the i-th fourth visual feature and the j-th third text feature. j ·T i It can represent the dot product between the j-th fourth visual feature and the i-th third text feature.

[0196] Furthermore, to improve the success rate of object detection and enhance its generalization ability by matching the directional position information in the speech information with the position information of entities in the target region, the initial object detection model may further include an initial feature extraction layer and an initial pixel coordinate encoding layer in some embodiments. Based on this, the above-mentioned extraction of visual features from the first image sample using the initial visual encoder in the initial object detection model to obtain the fourth visual feature may specifically include:

[0197] The channel information of each pixel in the first image sample is extracted using the initial feature extraction layer to obtain the first feature map sample;

[0198] Obtain the coordinate information of each pixel in the first image sample;

[0199] Using the initial pixel coordinate encoding layer, for each pixel, the coordinate information sample is added to the channel information of the first feature map sample to obtain the second feature map sample;

[0200] The visual features of the second feature map sample are extracted using the initial visual encoder to obtain the fourth visual feature.

[0201] The processing of the first image sample and the first text description sample by the initial target detection model in this embodiment can be found in the processing of the first image, the second image, and the text information by the target detection model described above, and will not be repeated in detail here.

[0202] Furthermore, as mentioned above, due to the hardware limitations of smart wearable devices and the need for fast response times in user interaction, excessive computational resources and long inference times required by object detection models can significantly impact the user experience. Therefore, to improve the efficiency of object detection and enhance the user experience of smart wearable devices, in some embodiments, after obtaining the object detection model, model distillation can be performed to reduce the model size, the number of parameters, the number of layers, and the width of the model.

[0203] Based on this, target detection methods can also include the specific process of model distillation, as shown below:

[0204] Obtain multiple second image samples segmented from the original image samples and a second text description sample corresponding to each second image sample;

[0205] The first visual encoder and the first text encoder in the object detection model are used as teacher models;

[0206] Obtain the student model corresponding to the teacher model. The student model includes a second visual encoder corresponding to the first visual encoder and a second text encoder corresponding to the first text encoder. The parameter value of the target parameter of the second visual encoder is less than the parameter value of the target parameter of the first visual encoder, and the parameter value of the target parameter of the second text encoder is less than the parameter value of the target parameter of the first text encoder.

[0207] The visual features of the second image sample are extracted using the first visual encoder to obtain the fifth visual feature;

[0208] The visual features of the second image sample are extracted using the second visual encoder to obtain the sixth visual features;

[0209] The text features of the second text description sample are extracted using the first text encoder to obtain the fourth text features;

[0210] The text features of the second text description sample are extracted using the second text encoder to obtain the fifth text features;

[0211] The second loss function value is determined based on the similarity between the sixth visual feature and the fifth text feature, the similarity between the sixth visual feature and the fifth visual feature, and the similarity between the fifth text feature and the fourth text feature;

[0212] If the second loss function value is greater than or equal to the second preset threshold, adjust the model parameters of the student model and return to the execution of extracting visual features of the second image sample using the first visual encoder until the second loss function value is less than the second preset threshold, and then determine the student model as the object detection model.

[0213] Here, the second image sample and its corresponding second text description sample may be the same as or different from the first image sample and its corresponding first text description sample. Additionally, the target parameters may include at least one of the following: the number of model parameters, model size, number of model layers, and width.

[0214] Additionally, the student model may also include a feature extraction layer and a pixel coordinate encoding layer. During model distillation, the student model uses a second visual encoder to extract visual features from the second image sample, obtaining a sixth visual feature, which may specifically include:

[0215] The channel information of each pixel in the second image sample is extracted using the feature extraction layer in the student model to obtain the third feature map sample;

[0216] Obtain the coordinate information of each pixel in the second image sample;

[0217] By using the pixel coordinate encoding layer in the student model, for each pixel, the coordinate information sample is added to the channel information of the third feature map sample to obtain the fourth feature map sample.

[0218] The visual features of the fourth feature map sample are extracted using the second visual encoder to obtain the sixth visual feature.

[0219] Similarly, during model distillation, the teacher model uses the first visual encoder to extract the visual features of the second image sample to obtain the fifth visual feature, which may include:

[0220] The channel information of each pixel in the second image sample is extracted using the feature extraction layer in the teacher model to obtain the fifth feature map sample;

[0221] Obtain the coordinate information of each pixel in the second image sample;

[0222] Using the pixel coordinate encoding layer in the teacher model, for each pixel, the coordinate information sample is added to the channel information of the fifth feature map sample to obtain the sixth feature map sample;

[0223] The visual features of the sixth feature map sample are extracted using the first visual encoder to obtain the fifth visual feature.

[0224] This application embodiment reduces the size of the target detection model by performing model distillation on the trained target detection model, thereby reducing the number of parameters, layers, and width of the target detection model, thus reducing the computational resources and inference time required for the target detection model to perform target detection, and improving the target detection efficiency.

[0225] To better understand the above solutions, some specific examples are given based on the above embodiments.

[0226] For example, such as Figure 3 As shown in the figure, a flowchart of target detection provided in this application embodiment may include the following steps:

[0227] S31, Receive the user's first instruction to capture an image;

[0228] S32. In response to the first instruction, capture a first image within the user's current field of view, and capture the area of ​​the user's gaze point to obtain a third image;

[0229] S33. Determine the target regions corresponding to each of the multiple entity objects in the first image;

[0230] S34. Cropping the first image according to multiple target regions yields multiple second images;

[0231] S35. Extract the visual features of the third image and multiple second images respectively;

[0232] S36. Obtain the user's voice information;

[0233] S37. Convert speech information into text information;

[0234] S38. Determine whether the text information includes the name of the target object. If not, execute S39; otherwise, execute S310.

[0235] S39. Based on the dialogue information between the user and the smart wearable device within a preset time period, determine the target object corresponding to the target pronoun in the text information;

[0236] S310. Extract the text features of the target object;

[0237] S311, Feature fusion and feature matching;

[0238] S312. Detect the target object based on the feature matching results.

[0239] In the embodiments of this application, steps S36-S310 may be located before any one of steps S31-S35, and no limitation is made here.

[0240] Furthermore, the above steps can be executed collaboratively by a camera module, object detection model, automatic speech recognition module, and natural language understanding module deployed in a smart wearable device. The object detection model may include a feature extraction layer, a region proposal network layer, a visual encoder, and a text encoder. Based on this, the schematic diagram of object detection provided in this application embodiment can... Figure 4 As shown.

[0241] Furthermore, as mentioned above, the smart wearable device in this application embodiment can be integrated with online e-commerce. Based on this, to improve the user experience of the smart wearable device, in some embodiments, after detecting the target object, the method may further include:

[0242] Retrieve the list of product links corresponding to the target object;

[0243] Display the target product link list based on the product link list;

[0244] Receive the user's first input when selecting a target product link from the list of target product links;

[0245] In response to the first input, create a target order corresponding to the target product link;

[0246] Receive the user's second input for payment of the target order;

[0247] In response to the second input, pay for the target order.

[0248] Here, the first and second inputs include, but are not limited to: touch input from the user using a touch device such as a finger or stylus to the target control, voice commands input by the user, specific gestures input by the user, and other feasible inputs. The specific first input can be determined according to actual usage needs, and this application embodiment does not limit it. The specific gesture in this application embodiment can be any one of a single-click gesture, a swipe gesture, a drag gesture, a pressure recognition gesture, a long-press gesture, an area change gesture, a double-press gesture, or a double-tap gesture; the click input in this application embodiment can be a single-click input, a double-tap input, or any number of clicks, and can also be a long-press input or a short-press input. Based on this, the first input mentioned above can be, for example, the user's click input on the target product link. The second input mentioned above can be, for example, the click input on the control used to confirm payment for the target order.

[0249] In addition, the target product link list can be the same as the product link list, or it can be a selection of product links obtained by filtering the product link list.

[0250] As an example, obtaining a list of product links corresponding to a target object could involve calling the search interface of an e-commerce platform to search for a list of product links corresponding to the target object. Specifically, a smart wearable device could call the search interface of an e-commerce platform, upload a second image including the target object to the e-commerce platform, and search for multiple links to products that are the same as or similar to the target object, thus obtaining a list of product links.

[0251] After displaying the list of target product links, users can select the desired item on the smart wearable device by clicking the target product link. The smart wearable device then responds to the user's action by creating a target order corresponding to the target product link. Users can instruct the smart wearable device to automatically place the order using voice or keyboard shortcuts, such as "Order the second item for me." After the order is successfully created, the user can confirm payment via voice or keyboard input, and the smart wearable device will complete the order payment upon receiving the instruction.

[0252] All steps in the embodiments of this application can be performed by smart wearable devices, eliminating the need for cross-device operations, simplifying the user's purchase process, and improving the user's experience with smart wearable devices.

[0253] Furthermore, to further enhance the user experience of smart wearable devices, in some embodiments, the above-mentioned display of the target product link list based on the product link list may specifically include:

[0254] Obtain user purchase preference information and product information for multiple products in the product link list;

[0255] Based on purchase preference information and product information, select multiple target products that the user is interested in from a variety of products;

[0256] Based on multiple target products, determine the list of target product links;

[0257] Displays a list of links to the target products.

[0258] Here, product information can include product ratings, store ratings, sales volume, and price. Additionally, smart wearable devices can pre-store user purchase preference information. This preference information can indicate whether a user prefers to buy products with higher ratings or higher sales volume.

[0259] As an example, after obtaining a list of product links, a smart wearable device can select multiple target products that the user is interested in from among the multiple products based on the product information corresponding to each product in the list and the user's purchase preference information, determine the target product link list, and display the target product link list.

[0260] This application embodiment first filters the list of product links to obtain the target product link list, and then displays the target product link list. This not only reduces the number of product link lists displayed, but also displays product links that users are more interested in, further improving the user experience of smart wearable devices.

[0261] Based on the above embodiments, the flowchart of purchasing the same product offline using a smart wearable device provided in this application embodiment can be as follows: Figure 5 As shown.

[0262] like Figure 5 As shown, purchasing the same product offline using a smart wearable device can include the following steps:

[0263] S51. The user sends a product identification and search command to the smart wearable device;

[0264] S52, The user issues a photo-taking command to the smart wearable device;

[0265] S53, The smart wearable device responds to the user's command, captures a first image, and detects the target object from the first image;

[0266] S54. The smart wearable device searches and sends images of the target object to the e-commerce platform;

[0267] S55. Search for the list of product links corresponding to the target object on the e-commerce platform;

[0268] S56. The e-commerce platform returns a list of product links corresponding to the target object to the smart wearable device;

[0269] S57. Smart wearable devices display a list of target product links based on a product link list;

[0270] S58. The user issues a purchase order to the smart wearable device.

[0271] S59. The smart wearable device responds to the purchase instruction and instructs the e-commerce platform to create a transaction order;

[0272] S510, e-commerce platform creates transaction orders;

[0273] S511. After an order is successfully created, the e-commerce platform returns a transaction serial number to the smart wearable device.

[0274] S512, Smart wearable devices display payment page;

[0275] S513, The user issues a payment instruction to the smart wearable device;

[0276] S514. After verifying the user's identity, the smart wearable device initiates a payment request to the e-commerce platform.

[0277] S515. The e-commerce platform responds to the payment request, executes the payment process, and obtains the payment result.

[0278] S516. E-commerce platforms return payment results to smart wearable devices;

[0279] S517, Smart wearable devices display payment results.

[0280] The embodiments of this application include at least the following technical points and effects:

[0281] 1) Achieve cross-modal extraction and fusion of natural language features and visual features: Based on the visual and voice sensors of smart wearable devices, receive the user's voice input and visual input, and extract and fuse features through deep learning models;

[0282] 2) Achieve the integration of object position weight information in the image: Utilize the advantages of images captured from the human eye's perspective by smart wearable devices and eye-tracking analysis to improve the accuracy of target detection;

[0283] 3) Achieve zero-shot referential object detection: When the object detection model has no training data for the target category, it can locate a specific target object in the image based on the user's verbal description, and has a strong generalization ability.

[0284] In summary, the multimodal fusion referential target detection method for smart wearable devices proposed in this application can detect target objects of interest to users when they wear smart wearable devices by using speech recognition, image recognition, and multimodal fusion matching between speech and images, and can achieve convenient online payment by combining smart payment capabilities.

[0285] Specifically, taking visual e-commerce scenarios as an example, users can scan offline physical items using smart wearable devices. The system can automatically identify item information and find similar products in real time, helping users quickly find the best way to buy and supporting one-click ordering, achieving "what you see is what you get." Smart wearable devices can identify and search for specific items or scenes within the user's field of vision, recommending related themed derivative products to meet the user's interests and needs.

[0286] It should be noted that the application scenarios described in the above embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0287] Based on the target detection method for wearable smart devices provided in the above embodiments, this application also provides specific implementations of a target detection device for wearable smart devices. Please refer to the following embodiments.

[0288] like Figure 6 As shown, the target detection device 600 provided in this application embodiment includes the following modules:

[0289] The first acquisition module 610 is used to acquire voice information output by the user and a first image captured by the smart wearable device, wherein the voice information includes information about the target object;

[0290] The first determining module 620 is used to determine the target regions corresponding to each of the multiple entity objects in the first image;

[0291] The cropping module 630 is used to crop the first image according to multiple target regions to obtain multiple second images;

[0292] The first extraction module 640 is used to extract visual features corresponding to the plurality of second images respectively to obtain a plurality of first visual features, and to extract text features of the target object from the speech information to obtain first text features;

[0293] The detection module 650 is used to detect the target object based on the second image corresponding to the target visual feature that satisfies a preset similarity condition with the first text feature among the plurality of first visual features.

[0294] The target detection device in this embodiment of the application determines target regions corresponding to multiple entity objects in a first image captured by a smart wearable device, and crops the first image according to the multiple target regions to obtain multiple second images. This segmentation of the complete first image into multiple second images that may include target objects is achieved. By extracting visual features corresponding to the multiple second images to obtain multiple first visual features, and extracting text features of the target object from the voice information to obtain first text features, and based on the second image corresponding to the target visual features that satisfy preset similarity conditions with the first text features, the target object is detected. This enables the detection of specific target objects from the entire image according to user intent, thereby improving the user experience of the smart wearable device.

[0295] The target detection device 600 described above will be explained in detail below:

[0296] To improve the efficiency of target region determination and adapt to smart wearable devices, in some embodiments, the smart wearable device deploys a target detection model, which includes a feature extraction layer and a region proposal network layer. Based on this, the first determination module 620 may specifically include:

[0297] The first extraction submodule is used to extract the channel information of each pixel in the first image using the feature extraction layer to obtain a first feature map;

[0298] A generation submodule is used to generate multiple candidate regions based on the first feature map using the region proposal network layer, and to predict the probability that the candidate regions include entity objects.

[0299] The first determining submodule is used to determine the candidate region corresponding to the probability greater than a preset probability threshold among the multiple probabilities as the target region.

[0300] To improve the generalization ability and accuracy of subsequent object detection, in some embodiments, the object detection model further includes a pixel coordinate encoding layer. Based on this, the first determining module 620 may further include:

[0301] The first acquisition submodule is used to acquire the coordinate information of each pixel in the first image before generating multiple candidate regions based on the first feature map using the region proposal network layer;

[0302] The first addition submodule is used to add the coordinate information of each pixel to the channel information of the first feature map using the pixel coordinate encoding layer, thereby obtaining the second feature map.

[0303] Based on this, the generated submodules may specifically include:

[0304] The generation unit is used to generate multiple candidate regions based on the second feature map using the region proposal network layer, and to predict the probability that the candidate regions include entity objects.

[0305] To improve the accuracy of target detection, in some embodiments, the detection module 650 may specifically include:

[0306] The mapping submodule is used to map the first text feature and multiple first visual features to the same vector space to obtain the second text feature and multiple second visual features;

[0307] The first detection submodule is used to detect the target object based on the second image corresponding to the target visual feature that satisfies the preset similarity condition of the second text feature among the plurality of second visual features.

[0308] To further improve the accuracy of target detection, in some embodiments, the target detection device 600 may further include:

[0309] The shooting module is used to capture a second image corresponding to a target visual feature that satisfies a preset similarity condition with the first text feature among the plurality of first visual features, and before detecting the target object, to capture the user's gaze point region to obtain a third image;

[0310] The second extraction module is used to extract the visual features of the third image to obtain the third visual features.

[0311] Based on this, the detection module 650 may specifically include:

[0312] The second detection submodule is used to detect the target object based on the second image corresponding to the target visual feature that satisfies the preset similarity condition of the first text feature among the plurality of first visual features and the third visual features.

[0313] To extract text features of a target object from speech information, in some embodiments, the target detection device 600 may further include:

[0314] The recognition module is used to extract the text features of the target object from the speech information. Before obtaining the first text features, the module performs automatic speech recognition on the speech information to obtain the text information.

[0315] The parsing module is used to perform semantic parsing on the text information to determine the target object.

[0316] Based on this, the first extraction module 640 may specifically include:

[0317] The second extraction submodule is used to extract the text features of the target object from the text information to obtain the first text feature.

[0318] To improve the generalization ability of object detection, in some embodiments, the speech information includes information on target pronouns for referring to the target object.

[0319] Based on this, the target detection device 600 may further include:

[0320] The second acquisition module is used to acquire dialogue information between the user and the smart wearable device within a preset time period before performing semantic parsing on the text information and determining the target object.

[0321] The third extraction module is used to extract entity objects from the dialogue information according to the order in which entity objects appear in the dialogue information.

[0322] Based on this, the parsing module may specifically include:

[0323] The parsing submodule is used to perform semantic parsing on the text information to determine the target pronoun;

[0324] The second determining submodule is used to determine the latest extracted entity object as the target object corresponding to the target pronoun.

[0325] In some embodiments, the smart wearable device deploys an object detection model, which further includes a visual encoder and a text encoder. Based on this, the first extraction module 640 may specifically include:

[0326] The third extraction submodule is used to extract the visual features corresponding to the plurality of second images using the visual encoder to obtain the plurality of first visual features;

[0327] The fourth extraction submodule is used to extract the text features of the target object from the text information using the text encoder to obtain the first text feature.

[0328] To improve the accuracy of target detection, in some embodiments, the detection module 650 may specifically include:

[0329] The second acquisition submodule is used to acquire the scene type of the first image and the object category of the entity objects in the second image corresponding to the multiple target visual features when there are multiple target visual features;

[0330] The third detection submodule is used to detect the target object based on the second image corresponding to the object category that matches the scene type.

[0331] To obtain the above-mentioned target detection model, in some embodiments, the target detection device 600 may further include:

[0332] The third acquisition module is used to acquire multiple first image samples segmented from the original image samples and a first text description sample corresponding to each first image sample;

[0333] The fourth extraction module is used to extract the visual features of the first image sample using the initial visual encoder in the initial target detection model, and obtain the fourth visual features;

[0334] The fifth extraction module is used to extract the text features of the first text description sample using the initial text encoder in the initial target detection model to obtain the third text features;

[0335] The second determining module is used to determine the value of the first loss function based on the similarity between the third text feature and the fourth visual feature by using a contrastive loss function. The contrastive loss function is used to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs.

[0336] The first training module is used to adjust the model parameters of the initial object detection model when the first loss function value is greater than or equal to the first preset threshold, and return to execute the step of extracting visual features of the first image sample using the initial visual encoder in the initial object detection model until the first loss function value is less than the first preset threshold, and determine the initial object detection model as the object detection model.

[0337] To improve the success rate of object detection and enhance its generalization ability by matching the directional position information in the speech information with the position information of entities in the target region, the initial object detection model in some embodiments further includes an initial feature extraction layer and an initial pixel coordinate encoding layer. Based on this, the fourth extraction module may specifically include:

[0338] The fifth extraction submodule is used to extract the channel information of each pixel in the first image sample using the initial feature extraction layer to obtain the first feature map sample;

[0339] The third acquisition submodule is used to acquire the coordinate information sample of each pixel in the first image sample;

[0340] The second addition submodule is used to add the coordinate information sample to the channel information of the first feature map sample for each pixel using the initial pixel coordinate encoding layer, so as to obtain the second feature map sample.

[0341] The sixth extraction submodule is used to extract the visual features of the second feature map sample using the initial visual encoder to obtain the fourth visual feature.

[0342] To improve the efficiency of target detection and enhance the user experience of smart wearable devices, in some embodiments, the target detection device 600 may further include:

[0343] The fourth acquisition module is used to acquire multiple second image samples segmented from the original image samples and a second text description sample corresponding to each second image sample;

[0344] The third determining module is used to use the first visual encoder and the first text encoder in the target detection model as teacher models;

[0345] The fifth acquisition module is used to acquire the student model corresponding to the teacher model. The student model includes a second visual encoder corresponding to the first visual encoder and a second text encoder corresponding to the first text encoder. The parameter value of the target parameter of the second visual encoder is less than the parameter value of the target parameter of the first visual encoder, and the parameter value of the target parameter of the second text encoder is less than the parameter value of the target parameter of the first text encoder.

[0346] The sixth extraction module is used to extract the visual features of the second image sample using the first visual encoder to obtain the fifth visual feature;

[0347] The seventh extraction module is used to extract the visual features of the second image sample using the second visual encoder to obtain the sixth visual feature;

[0348] The eighth extraction module is used to extract the text features of the second text description sample using the first text encoder to obtain the fourth text feature;

[0349] The ninth extraction module is used to extract the text features of the second text description sample using the second text encoder to obtain the fifth text feature;

[0350] The fourth determining module is used to determine the second loss function value based on the similarity between the sixth visual feature and the fifth text feature, the similarity between the sixth visual feature and the fifth visual feature, and the similarity between the fifth text feature and the fourth text feature;

[0351] The second training module is used to adjust the model parameters of the student model when the value of the second loss function is greater than or equal to the second preset threshold, and return to execute the step of extracting the visual features of the second image sample using the first visual encoder until the value of the second loss function is less than the second preset threshold, and determine the student model as the target detection model.

[0352] To enhance the user experience of smart wearable devices, in some embodiments, the target detection device 600 may further include:

[0353] The sixth acquisition module is used to acquire a list of product links corresponding to the target object after the target object is detected;

[0354] The display module is used to display the target product link list based on the product link list;

[0355] The receiving module is used to receive the user's first input of selecting a target product link from the list of target product links;

[0356] A creation module is used to create a target order corresponding to the target product link in response to the first input;

[0357] The receiving module is also used to receive a second input from the user for payment of the target order;

[0358] The payment module is used to pay the target order in response to the second input.

[0359] In some embodiments, the sixth acquisition module may specifically include:

[0360] The search submodule is used to call the search interface of the e-commerce platform to search for a list of product links corresponding to the target object.

[0361] To further enhance the user experience of smart wearable devices, in some embodiments, the display module may specifically include:

[0362] The fourth acquisition submodule is used to acquire the user's purchase preference information and the product information corresponding to multiple products in the product link list;

[0363] The selection submodule is used to select multiple target products that the user is interested in from the multiple products based on the purchase preference information and the product information.

[0364] The third determining submodule is used to determine a list of target product links based on the multiple target products;

[0365] The display submodule is used to display the list of links to the target products.

[0366] Based on the target detection method provided in the above embodiments, this application also provides specific implementation methods for electronic devices. Figure 7 A schematic diagram of an electronic device 700 provided in an embodiment of this application is shown.

[0367] Electronic device 700 may include processor 710 and memory 720 storing computer program instructions.

[0368] Specifically, the processor 710 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0369] Memory 720 may include mass storage for data or instructions. For example, and not limitingly, memory 720 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where suitable, memory 720 may include removable or non-removable (or fixed) media. Where suitable, memory 720 may be internal or external to electronic device 700. In a particular embodiment, memory 720 is a non-volatile solid-state memory.

[0370] In a specific embodiment, the memory 720 can be implemented as ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 720 and executed by the processor 710. The processor 710 implements any of the target detection methods in the above embodiments by reading and executing the computer program instructions stored in the memory 720.

[0371] The processor 710 implements any of the target detection methods described in the above embodiments by reading and executing computer program instructions stored in the memory 720.

[0372] In one example, the electronic device 700 may further include a communication interface 730 and a bus 740. For example, Figure 7 As shown, the processor 710, memory 720, and communication interface 730 are connected via bus 740 and communicate with each other.

[0373] The communication interface 730 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0374] Bus 740 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 740 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0375] For example, the electronic device 700 can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0376] The electronic device can execute the target detection method in the embodiments of this application, thereby achieving a combination Figures 1 to 5 The target detection method described herein, and the beneficial effects of the corresponding method embodiments, will not be elaborated further here.

[0377] Furthermore, in conjunction with the target detection methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the target detection methods in the above embodiments.

[0378] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0379] The computer program instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the target detection method as shown in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0380] Based on the target detection methods described in the above embodiments, this application provides a computer program product for implementation. When the instructions in this computer program product are executed by the processor of an electronic device, they implement any of the target detection methods described in the above embodiments.

[0381] The computer program products of the above embodiments are used to implement the target detection method shown in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0382] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0383] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0384] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0385] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0386] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A target detection method, characterized in that, Applied to smart wearable devices, the method includes: Acquire user-output voice information and a first image captured by the smart wearable device, wherein the voice information includes information about the target object; In the first image, the target regions corresponding to each of the multiple entity objects are determined; The first image is cropped according to multiple target regions to obtain multiple second images; Visual features corresponding to the multiple second images are extracted to obtain multiple first visual features, and text features of the target object are extracted from the speech information to obtain first text features; The target object is detected based on the second image corresponding to the target visual feature that satisfies the preset similarity condition of the first text feature among the plurality of first visual features.

2. The method according to claim 1, characterized in that, The smart wearable device is equipped with an object detection model, which includes a feature extraction layer and a region proposal network layer. The step of determining the target regions corresponding to multiple entity objects in the first image includes: The feature extraction layer is used to extract the channel information of each pixel in the first image to obtain a first feature map. Using the region proposal network layer, multiple candidate regions are generated based on the first feature map, and the probability that the candidate regions include entity objects is predicted. The candidate regions corresponding to the probabilities greater than a preset probability threshold among the multiple probabilities are determined as the target regions.

3. The method according to claim 2, characterized in that, The target detection model further includes a pixel coordinate encoding layer. Before generating multiple candidate regions based on the first feature map using the region proposal network layer and predicting the probability that the candidate regions contain entity objects, the method further includes: Obtain the coordinate information of each pixel in the first image; Using the pixel coordinate encoding layer, for each pixel, the coordinate information is added to the channel information of the first feature map to obtain the second feature map; The step of using the region proposal network layer to generate multiple candidate regions based on the first feature map and predicting the probability that the candidate regions contain entity objects includes: Using the region proposal network layer, multiple candidate regions are generated based on the second feature map, and the probability that the candidate regions include entity objects is predicted.

4. The method according to claim 1, characterized in that, The method of detecting the target object based on the second image corresponding to the target visual feature that satisfies a preset similarity condition with the first text feature among the plurality of first visual features includes: The first text feature and multiple first visual features are mapped to the same vector space to obtain the second text feature and multiple second visual features; The target object is detected based on the second image corresponding to the target visual feature that satisfies the preset similarity condition of the second text feature among the plurality of second visual features.

5. The method according to claim 1, characterized in that, Before detecting the target object based on the second image corresponding to the target visual feature that satisfies a preset similarity condition with the first text feature among the plurality of first visual features, the method further includes: A third image is obtained by capturing the user's gaze point area; Extract the visual features of the third image to obtain the third visual features; The method of detecting the target object based on the second image corresponding to the target visual feature that satisfies a preset similarity condition with the first text feature among the plurality of first visual features includes: The target object is detected based on the second image corresponding to the target visual feature among the plurality of first visual features and the third visual features that satisfies the preset similarity condition with the first text features.

6. The method according to claim 1, characterized in that, Before extracting the text features of the target object from the speech information to obtain the first text features, the method further includes: The voice information is automatically recognized to obtain text information; The text information is semantically parsed to determine the target object; The step of extracting text features of the target object from the speech information to obtain the first text feature includes: The text features of the target object are extracted from the text information to obtain the first text feature.

7. The method according to claim 6, characterized in that, The speech information includes information about target pronouns used to refer to the target object. Before performing semantic parsing on the text information to determine the target object, the method further includes: Acquire dialogue information between the user and the smart wearable device within a preset time period; Extract entity objects from the dialogue information according to the order in which they appear. The step of semantically parsing the text information to determine the target object includes: The text information is semantically parsed to determine the target pronoun; The newly extracted entity object is identified as the target object corresponding to the target pronoun.

8. The method according to claim 6, characterized in that, The smart wearable device is equipped with a target detection model, which further includes a visual encoder and a text encoder. The model extracts visual features corresponding to the multiple second images to obtain multiple first visual features, and extracts text features of the target object from the speech information to obtain first text features, including: The visual encoder is used to extract the visual features corresponding to the plurality of second images respectively to obtain the plurality of first visual features; Using the text encoder, the text features of the target object are extracted from the text information to obtain the first text feature.

9. The method according to any one of claims 1-8, characterized in that, The method of detecting the target object based on the second image corresponding to the target visual feature that satisfies a preset similarity condition with the first text feature among the plurality of first visual features includes: When there are multiple target visual features, the scene type of the first image and the object category of the entity objects in the second image corresponding to the multiple target visual features are obtained; The target object is detected based on the second image corresponding to the object category that matches the scene type.

10. The method according to any one of claims 1-8, characterized in that, The method further includes: Obtain multiple first image samples segmented from the original image samples and a first text description sample corresponding to each first image sample; The visual features of the first image sample are extracted using the initial visual encoder in the initial object detection model to obtain the fourth visual features; The text features of the first text description sample are extracted using the initial text encoder in the initial target detection model to obtain the third text features; By comparing loss functions, the value of the first loss function is determined based on the similarity between the third text feature and the fourth visual feature. The comparison loss function is used to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs. If the first loss function value is greater than or equal to the first preset threshold, the model parameters of the initial object detection model are adjusted, and the process of extracting visual features of the first image sample using the initial visual encoder in the initial object detection model is repeated until the first loss function value is less than the first preset threshold, and the initial object detection model is determined as the object detection model.

11. The method according to claim 10, characterized in that, The initial object detection model further includes an initial feature extraction layer and an initial pixel coordinate encoding layer. The initial visual encoder in the initial object detection model extracts the visual features of the first image sample to obtain the fourth visual feature, including: The channel information of each pixel in the first image sample is extracted using the initial feature extraction layer to obtain the first feature map sample; Obtain the coordinate information of each pixel in the first image sample; Using the initial pixel coordinate encoding layer, for each pixel, the coordinate information sample is added to the channel information of the first feature map sample to obtain the second feature map sample; The visual features of the second feature map sample are extracted using the initial visual encoder to obtain the fourth visual feature.

12. The method according to claim 10, characterized in that, After obtaining the target detection model, the method further includes: Obtain multiple second image samples segmented from the original image samples and a second text description sample corresponding to each second image sample; The first visual encoder and the first text encoder in the object detection model are used as teacher models; Obtain a student model corresponding to the teacher model. The student model includes a second visual encoder corresponding to the first visual encoder and a second text encoder corresponding to the first text encoder. The parameter value of the target parameter of the second visual encoder is less than the parameter value of the target parameter of the first visual encoder, and the parameter value of the target parameter of the second text encoder is less than the parameter value of the target parameter of the first text encoder. The visual features of the second image sample are extracted using the first visual encoder to obtain the fifth visual feature; The visual features of the second image sample are extracted using the second visual encoder to obtain the sixth visual feature; The first text encoder is used to extract the text features of the second text description sample to obtain the fourth text features; The text features of the second text description sample are extracted using the second text encoder to obtain the fifth text feature; The second loss function value is determined based on the similarity between the sixth visual feature and the fifth text feature, the similarity between the sixth visual feature and the fifth visual feature, and the similarity between the fifth text feature and the fourth text feature; If the value of the second loss function is greater than or equal to the second preset threshold, the model parameters of the student model are adjusted, and the process of extracting the visual features of the second image sample using the first visual encoder is repeated until the value of the second loss function is less than the second preset threshold, at which point the student model is determined as the target detection model.

13. The method according to any one of claims 1-8, characterized in that, After detecting the target object, the method further includes: Retrieve a list of product links corresponding to the target object; Based on the product link list, display the target product link list; Receive the user's first input to select a target product link from the list of target product links; In response to the first input, a target order corresponding to the target product link is created; Receive a second input from the user for payment of the target order; In response to the second input, the target order is paid.

14. The method according to claim 13, characterized in that, The step of obtaining the list of product links corresponding to the target object includes: Call the e-commerce platform's search interface to search for a list of product links corresponding to the target object.

15. The method according to claim 13, characterized in that, The step of displaying the target product link list based on the product link list includes: Obtain the user's purchase preference information and the product information corresponding to multiple products in the product link list; Based on the purchase preference information and the product information, select multiple target products that the user is interested in from the plurality of products; Based on the multiple target products, a list of target product links is determined; Display the list of links to the target product.

16. A target detection device, characterized in that, The device is used in smart wearable devices and includes: The first acquisition module is used to acquire voice information output by the user and a first image captured by the smart wearable device, wherein the voice information includes information about the target object; The first determining module is used to determine the target regions corresponding to each of the multiple entity objects in the first image; A cropping module is used to crop the first image according to multiple target regions to obtain multiple second images; The first extraction module is used to extract visual features corresponding to the plurality of second images respectively to obtain a plurality of first visual features, and to extract text features of the target object from the speech information to obtain first text features; The detection module is used to detect the target object based on the second image corresponding to the target visual feature that satisfies a preset similarity condition with the first text feature among the plurality of first visual features.

17. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the target detection method as described in any one of claims 1-15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the target detection method as described in any one of claims 1-15.

19. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the target detection method as described in any one of claims 1-15.

Images