Target positioning method and device based on large model, electronic equipment and storage medium

Abstract

The invention discloses a target positioning method and device based on a large model, electronic equipment and a storage medium, and relates to the technical field of computers, in particular to the fields of large models, voice technologies, computer vision, deep learning and the like. According to the scheme, a positioning request sent by a target terminal is received, wherein the positioning request comprises a target picture and a voice instruction; extracting first object information of the to-be-positioned object from the voice instruction, and performing target detection on the target picture according to the first object information to obtain a detection result; intercepting an object picture of the candidate object from the target picture according to the position information of the candidate object in the target picture; determining a target object from the candidate objects by using a large model according to the object picture, the position information of the candidate objects in the target picture and the first object information; and sending the position information of the target object in the target picture to the terminal, so that the target terminal determines the position information of the target object relative to the target terminal according to the position information of the target object in the target picture.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to artificial intelligence fields such as large models, speech technology, computer vision, and deep learning, and specifically to a target positioning method, device, electronic device, and storage medium based on a large model. Background Art

[0002] Target localization technology is a key technology in computer vision. It can identify targets in images or video sequences and determine their precise positions in the images or scenes. Target localization technology is widely used in many fields, such as robot navigation, self-driving cars, video surveillance, etc. Summary of the invention

[0003] The present application provides a target positioning method, device, electronic device and storage medium based on a large model.

[0004] According to one aspect of the present application, a target positioning method based on a large model is provided, comprising:

[0005] Receiving a positioning request sent by a target terminal; wherein the positioning request includes a target image and a voice command;

[0006] Extracting first object information of the object to be located from the voice command, and performing object detection on the target image according to the first object information to obtain a detection result; wherein the detection result includes position information of the candidate object in the target image;

[0007] According to the position information of the candidate object in the target image, intercepting the object image of the candidate object from the target image;

[0008] Determine the target object from the candidate objects using a large model according to the object picture, the position information of the candidate object in the target picture and the first object information;

[0009] The position information of the target object in the target picture is sent to the terminal, so that the target terminal determines the position information of the target object relative to the target terminal according to the position information of the target object in the target picture.

[0010] According to another aspect of the present application, a target positioning device based on a large model is provided, comprising:

[0011] A receiving module, configured to receive a positioning request sent by a target terminal; wherein the positioning request includes a target image and a voice command;

[0012] An extraction module, used to extract first object information of the object to be located from the voice command;

[0013] A detection module, configured to perform target detection on the target image according to the first object information to obtain a detection result; wherein the detection result includes position information of the candidate object in the target image;

[0014] A screenshot module, configured to capture an object image of the candidate object from the target image according to position information of the candidate object in the target image;

[0015] A retrieval module, configured to determine a target object from the candidate objects using a large model according to the object image, the position information of the candidate object in the target image and the first object information;

[0016] The sending module is used to send the position information of the target object in the target picture to the terminal, so that the target terminal determines the position information of the target object relative to the target terminal according to the position information of the target object in the target picture.

[0017] According to another aspect of the present application, there is provided an electronic device, including:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the method described in the above embodiment.

[0021] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to the above embodiment.

[0022] According to another aspect of the present application, a computer program product is provided, including a computer program, wherein the computer program implements the steps of the method described in the above embodiment when executed by a processor.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0025] Figure 1A schematic diagram of a process of a target positioning method based on a large model provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of a flow chart of a target positioning method based on a large model provided in another embodiment of the present application;

[0027] Figure 3 A schematic diagram of a flow chart of a target positioning method based on a large model provided in another embodiment of the present application;

[0028] Figure 4 A schematic diagram of a process of a target positioning method based on a large model provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of a target positioning device based on a large model provided in one embodiment of the present application;

[0030] Figure 6 It is a block diagram of an electronic device used to implement the large model-based target positioning method of an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0032] The following describes the target positioning method, device, electronic device and storage medium based on a large model in embodiments of the present application with reference to the accompanying drawings.

[0033] Figure 1 A flowchart of a large model-based target positioning method provided in one embodiment of the present application.

[0034] The large model-based target positioning method of the embodiment of the present application can be executed by the large model-based target positioning device of the embodiment of the present application, and the device can be configured in an electronic device to realize the target positioning function.

[0035] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0036] like Figure 1 As shown, the target positioning method based on the large model includes:

[0037] Step 101: receiving a positioning request sent by a target terminal.

[0038] In the present application, the target terminal may include a camera device, which can be used to shoot scenes within the shooting range. In addition, the target terminal also has the function of recording and playing sounds, and the target terminal can interact with the user through voice. After the target terminal collects the user's voice, it can perform intent recognition on the collected voice to determine the user's intention and determine whether the user's intention contains target positioning requirements. If the user's intention contains target positioning requirements, a positioning request can be sent to the server, so that the server can receive the positioning request sent by the target terminal.

[0039] The positioning request may include a target image, a voice command, etc. collected by the target terminal. For example, the target image may be taken by the target terminal while collecting the user's voice, or taken after determining that the user's intention includes the target positioning requirement, etc. For example, the voice command may refer to a voice including the target positioning requirement.

[0040] For example, the user inputs the voice command "pass me the apple on the left", and after the target terminal recognizes the intention of the voice command and determines that the user's intention includes the need to locate the apple, it sends a positioning request to the server.

[0041] Exemplarily, the microphone of the target terminal may be always in an on state, and the user may directly input voice, or there may be a voice input control on the target terminal, and the user may input voice while triggering the voice input control.

[0042] Step 102: extract first object information of the object to be located from the voice command, and perform object detection on the target image according to the first object information to obtain a detection result.

[0043] The first object information may include the name of the object to be located, and may also include features such as color, shape, size, and position of the object to be located.

[0044] Exemplarily, voice recognition may be performed on the voice command to convert the voice command into text, and the text may be segmented to obtain segmented words, and part-of-speech tags may be performed on the segmented words. Named entity recognition may then be performed on the part-of-speech tagging results to obtain first object information.

[0045] In the present application, the detection result of the target image may include the position information of the candidate object in the target image, the detection score, etc.

[0046] Exemplarily, the position information of the candidate object in the target image may refer to the position information of the detection box of the candidate object in the target image, and the position information may include the coordinates of four vertices of the detection box in the target image.

[0047] The detection score can be used to represent the confidence of the position information of the detected candidate object. In addition, if there are multiple candidate objects, each candidate object can correspond to a detection score.

[0048] As a possible implementation, if the first object information includes the name of the object, feature extraction can be performed on the object name to obtain text features, feature extraction can be performed on the target image to obtain image features, and detection results can be obtained based on the text features and image features. Thus, target detection can be performed based on the image features extracted from the target image in combination with the text features extracted from the object name, which can improve the accuracy of the detection results and the detection efficiency.

[0049] For example, if the name of the object to be located is extracted from the voice command as apple, then target detection can be performed on the target image based on the object name to detect the apple from the target image and obtain the location information, detection score, etc. of the apple in the target image.

[0050] It should be noted that in the present application, the number of detected candidate objects can be one or more, and there is no limitation on this.

[0051] Step 103 : capturing an object image of the candidate object from the target image according to the position information of the candidate object in the target image.

[0052] In the present application, the target image can be captured according to the detection frame of the candidate object to obtain an object image of the candidate object.

[0053] Step 104 , determining the target object from the candidate objects using the large model according to the object image, the position information of the candidate object in the target image and the first object information.

[0054] In the present application, the object image of the candidate object, the position information of the candidate object in the target image, the first object information, etc. can be input into the large model, and the large model can be used to retrieve the target object to be finally located from the detected candidate objects.

[0055] Step 105 : sending the position information of the target object in the target picture to the terminal, so that the target terminal determines the position information of the target object relative to the target terminal according to the position information of the target object in the target picture.

[0056] In the present application, the position information of the target object in the target image can be sent to the target terminal. The target terminal determines the distance from the target object to the target terminal based on the position information of the target object in the target image. Based on the distance, the position information of the target object relative to the target terminal can be determined.

[0057] In an embodiment of the present application, the object information of the object to be located is extracted from the voice command sent by the target terminal, and according to the object information, the target image sent by the target terminal is subjected to target detection. Based on the detection result and the extracted object information, the target object to be located is determined by using a large model, and then the position information of the target object in the target image is sent to the target terminal, so that the target terminal determines the position of the target object relative to the target terminal based on the position information, thereby achieving the positioning of the target object. Thus, by extracting the information of the specified object to be located from the natural language, and then performing target detection, and using the large model to retrieve the object to be located from the detection result, the accuracy of target positioning can be improved. In addition, by combining the object information extracted from the natural language and performing target detection on the target image, the accuracy of the detection result can be improved.

[0058] Figure 2 A schematic flowchart of a large model-based target positioning method provided in another embodiment of the present application.

[0059] like Figure 2 As shown, the target positioning method based on the large model includes:

[0060] Step 201: receiving a positioning request sent by a target terminal.

[0061] In the present application, step 201 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.

[0062] Step 202: extract first object information of the object to be located from the voice command, and perform object detection on the target image according to the first object information to obtain a detection result.

[0063] Exemplarily, feature extraction can be performed on the first object information to obtain a first text feature, and feature extraction can be performed on the target image to obtain a first image feature. The first text feature and the first image feature can be fused using a self-attention mechanism, image-to-text cross-attention, and text-to-image cross-attention to obtain a second text feature and a second image feature. The target image feature with the highest match degree with the second text feature is determined among the second image features, and decoding is performed based on the target image feature to obtain a detection result.

[0064] For example, the target image feature may be used as a query of a decoder to decode the target image feature and obtain a detection result, wherein the query of the decoder may include a content query and a location query.

[0065] Therefore, using text features to guide the selection of image features and decoding based on the selected target image features can improve the accuracy of the detection results.

[0066] Step 203 : capturing an object image of the candidate object from the target image according to the position information of the candidate object in the target image.

[0067] In the present application, step 203 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.

[0068] Step 204 , obtaining prompt information according to the object picture, the position information of the candidate object in the target picture and the first object information.

[0069] In the present application, the preset prompt template may be filled according to the object image of the candidate object, the position information of the candidate object in the target image, the first object information, etc., to obtain the prompt information.

[0070] Among them, the prompt information can be used to prompt the large model to perform visual retrieval tasks.

[0071] Exemplarily, if the number of candidate objects exceeds a preset number, an object picture containing these candidate objects can be captured from the target picture, and prompt information can be generated according to the object picture, position information of each candidate object in the target picture, first object information, etc.

[0072] Exemplarily, if the number of candidates does not exceed a preset number, a picture of each candidate object can be captured from the target picture, and for each candidate object, prompt information corresponding to each candidate object is generated based on the object picture of each candidate object, the position information of the candidate object in the target picture, the first object information, etc.

[0073] Step 205: input the prompt information into the large model to determine whether the candidate object meets the positioning requirements.

[0074] In the present application, the prompt may be input into the large model, and the large model determines whether the candidate object meets the positioning requirement based on the prompt information, that is, determines whether the candidate object is the object that the user wants to locate. Exemplarily, the positioning requirement may include the first object information extracted from the voice command.

[0075] Exemplarily, in addition to the object image of the candidate object, the position information of the candidate object in the target image, the first object information, etc., the prompt information may also include large model output requirements, such as output format requirements. For example, outputting "1" indicates that the candidate object is the object that the user wants to locate, and outputting "0" indicates that the candidate object is not the object that the user wants to locate.

[0076] Exemplarily, the large model can be used to extract object information from the object image and the position information of the candidate object in the target image to obtain the second object information, and determine whether the candidate object meets the positioning requirements based on the matching between the second object information and the first object information. For example, if the second object information matches the first object information, it can be determined that the candidate object meets the positioning requirements, and if the second object information does not match the first object information, it can be determined that the candidate object does not meet the positioning requirements.

[0077] The second object information may include but is not limited to the name, color, shape, position, etc. of the candidate object. The position here may refer to the position of the candidate object in the target image.

[0078] Exemplarily, the first object information includes the position information of the object to be located. Based on the picture of the candidate object, it can be determined whether the position information of the candidate object in the detection result matches the position information of the object to be located in the first object information. If they match, it can be determined that the candidate object meets the positioning requirements, that is, the candidate object is the object that the user wants to locate. If they do not match, it can be determined that the candidate object is not the object that the user wants to locate.

[0079] Therefore, the large model is used to extract object information from the object image and the position information of the candidate object in the target image, and based on the match between the extracted second object information and the first object information extracted from the voice command, it is determined whether the candidate object is the object the user wants to locate, thereby using the large model for visual retrieval to improve the accuracy of the retrieval.

[0080] Step 206, determining the target object according to the judgment result of the large model.

[0081] In this application, based on the judgment results of the large model, it is determined which candidate objects meet the positioning requirements and the number of candidate objects that meet the positioning requirements. If there is a candidate object that meets the positioning requirements, the candidate object can be determined as the target object. If there are multiple candidate objects that meet the positioning requirements, the candidate object with the highest detection score among the multiple candidate objects can be determined as the target object.

[0082] Therefore, when the large model determines that there are multiple candidate objects that are the objects that the user wants to locate, the candidate object with the highest detection score can be used as the final object to be located, thereby improving the accuracy of object retrieval.

[0083] Step 207 : sending the position information of the target object in the target picture to the terminal, so that the target terminal determines the position information of the target object relative to the target terminal according to the position information of the target object in the target picture.

[0084] In the present application, step 207 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.

[0085] In an embodiment of the present application, prompt information is constructed based on the object image, the position information of the candidate object in the target image and the first object information, the prompt information is input into the large model, and the large model is used to perform visual retrieval to retrieve the object to be ultimately located from the candidate objects, thereby improving the accuracy of the retrieval.

[0086] Figure 3 A schematic flowchart of a large model-based target positioning method provided in another embodiment of the present application.

[0087] like Figure 3 As shown, the target positioning method based on the large model includes:

[0088] Step 301: receiving a positioning request sent by a target terminal.

[0089] In the present application, step 301 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.

[0090] Step 302: Perform voice recognition on the voice instruction to convert the voice instruction into text.

[0091] In the present application, a pre-trained speech recognition model may be used to perform speech recognition on the voice command to convert the voice command into text.

[0092] Step 303: Use the large model to extract object information from the text to obtain first object information.

[0093] In the present application, the corresponding prompt template can be filled in according to the text converted from the voice command to obtain the corresponding prompt information. The prompt information can be used to prompt the large model to perform the object information extraction task. The prompt information is input into the large model, and the large model is used to extract object information from the text to obtain the first object information.

[0094] Step 304: perform target detection on the target image according to the first object information to obtain a detection result.

[0095] Step 305 , capturing an object image of the candidate object from the target image according to the position information of the candidate object in the target image.

[0096] Step 306 , obtaining prompt information according to the object picture, the position information of the candidate object in the target picture and the first object information.

[0097] Step 307: input the prompt information into the large model to determine whether the candidate object meets the positioning requirements.

[0098] Step 308, determining the target object according to the judgment result of the large model.

[0099] In the present application, steps 304 to 308 may be implemented in any of the embodiments of the present application, and therefore will not be described in detail herein.

[0100] Step 309 : sending the position information of the target object in the target picture to the terminal, so that the target terminal determines the position information of the target object relative to the target terminal according to the position information of the target object in the target picture.

[0101] Exemplarily, the target terminal may include a distance measuring device, and the target terminal may use the distance measuring device to measure the distance between the target object and the target terminal based on the position information of the target object in the target picture, and determine the position information of the target object relative to the target terminal based on the distance and the position information of the target object in the target picture.

[0102] Thus, the target terminal can measure the distance between the target terminal and the target object through the distance measuring device, and then determine the position information of the target object relative to the target terminal, thereby achieving accurate positioning of the target object in the scene.

[0103] Exemplarily, the distance measuring device may be an infrared sensor, an ultrasonic device, etc., or may be other devices that can be used to measure the distance between objects, which is not limited.

[0104] For example, if there is an infrared sensor on the target terminal, the horizontal straight-line distance from the target object to the target terminal can be measured by the infrared sensor to finally obtain the coordinates of the target object relative to the target terminal.

[0105] Exemplarily, the target terminal may be a robot having a robotic arm, and the target terminal may also control the robotic arm to pick up the target object according to position information of the target object relative to the target terminal.

[0106] Therefore, the target terminal can be a robot. During the human-computer interaction process, the target can be located using a large model based on the voice commands input by the user. The robot can pick up objects based on the positioning results, thereby completing complex human-computer interaction behaviors and enriching the human-computer interaction experience.

[0107] In an embodiment of the present application, a large model can be used to extract the object to be located specified by the user from natural language, thereby improving the accuracy of information extraction, and the large model can be used to retrieve the final object to be located from the candidate objects through visual retrieval, thereby improving the accuracy of retrieval and further improving the accuracy of target positioning.

[0108] For ease of understanding, the following Figure 4 To explain, Figure 4 A process diagram of a large model-based target positioning method provided in an embodiment of the present application.

[0109] Taking the target terminal as a smart terminal as an example, the smart terminal can include a camera, which can capture the front image and transmit it to the cloud service. In addition, the camera also includes an infrared sensor that can measure the horizontal straight-line distance of objects in the camera image. At the same time, the smart terminal has the ability to record and play sound, and it can transmit the user's voice commands to the cloud service.

[0110] In the process of target positioning based on large models, the services used can be as follows:

[0111] Access service: Calling other services based on the user's location request;

[0112] Speech conversion service: a service that converts speech into text;

[0113] Extraction service: A service that extracts the name of the object (such as apple or clothes) and features (such as color, shape, and location) to be retrieved in the user's voice command, by calling the model service;

[0114] Object detection service: The input is the name of the object to be retrieved and the target image, and the output is the location of the object in the target image and a screenshot;

[0115] Retrieval service: Retrieve the object images determined by the target detection service, and use a large model (such as the image itself, location, etc.) to determine whether it is the object the user wants to retrieve. If multiple conditions are met, the object with the highest score from the target detection service is returned, that is, the object with the highest detection score.

[0116] Model services: services provided by large models.

[0117] like Figure 4 As shown in the figure, the target positioning process based on the large model is as follows:

[0118] The user transmits the voice command to the [smart terminal], and the [smart terminal] sends a positioning request through the http protocol. The service is triggered by the http trigger and the positioning request is transmitted to the [access service]. The [access service] calls the [voice conversion service] to convert the voice command in the positioning request into text, and then calls the [extraction service] to extract the object name and color, shape, position and other features in the text. Then, the [target detection service] is called to pass in the object name to obtain one or more detected object images, positions, detection scores and other information, and then the [retrieval service] is called for retrieval. The [retrieval service] can construct a prompt word based on the object image, the position information in the target image, the object features extracted by the extraction service, and the object name and send it to the [model service] to determine whether it is the object that the user wants to locate. If there are multiple objects, the position of the object with the highest score in the target image is returned by the [target detection service]. Then, the [smart terminal] measures the horizontal straight-line distance of the object in the determined target image by calling the infrared sensor, and finally obtains a coordinate relative to the [smart terminal].

[0119] In order to implement the above embodiment, the embodiment of the present application also proposes a target positioning device based on a large model. Figure 5 A schematic diagram of the structure of a target positioning device based on a large model provided in one embodiment of the present application.

[0120] like Figure 5 As shown, the target positioning device 500 based on the large model includes:

[0121] The receiving module 510 is used to receive a positioning request sent by a target terminal; wherein the positioning request includes a target image and a voice command;

[0122] An extraction module 520, configured to extract first object information of the object to be located from the voice command;

[0123] A detection module 530, configured to perform target detection on the target image according to the first object information to obtain a detection result; wherein the detection result includes position information of the candidate object in the target image;

[0124] A screenshot module 540, configured to capture an object image of the candidate object from the target image according to position information of the candidate object in the target image;

[0125] A retrieval module 550, configured to determine a target object from the candidate objects using a large model according to the object image, the position information of the candidate object in the target image and the first object information;

[0126] The sending module 560 is used to send the position information of the target object in the target picture to the terminal, so that the target terminal determines the position information of the target object relative to the target terminal according to the position information of the target object in the target picture.

[0127] Optionally, the retrieval module 550 is used to:

[0128] Acquire prompt information according to the object picture, the position information of the candidate object in the target picture and the first object information;

[0129] Inputting the prompt information into the large model to determine whether the candidate object meets the positioning requirements;

[0130] The target object is determined according to the judgment result of the large model.

[0131] Optionally, the detection result further includes a detection score, and the retrieval module 550 is used to:

[0132] In response to determining that there are multiple candidate objects that meet the positioning requirement according to the judgment result, a candidate object with the highest detection score among the multiple candidate objects is determined as the target object.

[0133] Optionally, the retrieval module 550 is used to:

[0134] Using the large model, extracting object information from the object image and the position information of the candidate object in the target image to obtain second object information;

[0135] According to the matching condition between the second object information and the first object information, it is determined whether the candidate object meets the positioning requirement.

[0136] Optionally, the extraction module 520 is used to:

[0137] Performing voice recognition on the voice command to convert the voice command into text;

[0138] The large model is used to extract object information from the text to obtain the first object information.

[0139] Optionally, the detection module 530 is used to:

[0140] Performing feature extraction on the first object information to obtain a first text feature;

[0141] Performing feature extraction on the target image to obtain a first image feature;

[0142] fusing the first text feature and the first image feature to obtain a second text feature and a second image feature;

[0143] Determining a target image feature having the highest matching degree with the second text feature among the second image features;

[0144] Decoding is performed according to the target image features to obtain the detection result.

[0145] Optionally, the target terminal includes a distance measuring device, and the target terminal uses the distance measuring device to measure the distance between the target object and the target terminal according to the position information of the target object in the target image; and determines the position information of the target object relative to the target terminal according to the distance.

[0146] Optionally, the target terminal is a robot having a mechanical arm, and the target terminal is further used to control the mechanical arm to pick up the target object according to position information of the target object relative to the target terminal.

[0147] It should be noted that the explanation of the aforementioned large-model-based target positioning method embodiment is also applicable to the large-model-based target positioning device of this embodiment, so it will not be repeated here.

[0148] In an embodiment of the present application, the object information of the object to be located is extracted from the voice command sent by the target terminal, and according to the object information, the target image sent by the target terminal is subjected to target detection. Based on the detection result and the extracted object information, the target object to be located is determined by using a large model, and then the position information of the target object in the target image is sent to the target terminal, so that the target terminal determines the position of the target object relative to the target terminal based on the position information, thereby achieving the positioning of the target object. Thus, by extracting the information of the specified object to be located from the natural language, and then performing target detection, and using the large model to retrieve the object to be located from the detection result, the accuracy of target positioning can be improved. In addition, by combining the object information extracted from the natural language and performing target detection on the target image, the accuracy of the detection result can be improved.

[0149] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0150] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0151] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 602 or a computer program loaded from a storage unit 608 to a RAM (Random Access Memory) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An I / O (Input / Output) interface 605 is also connected to the bus 604.

[0152] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0153] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as a target positioning method based on a large model. For example, in some embodiments, the target positioning method based on a large model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the target positioning method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute a large model-based target positioning method in any other appropriate manner (eg, by means of firmware).

[0154] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0156] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0158] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0159] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0160] According to an embodiment of the present application, the present application also provides a computer program product, which, when an instruction processor in the computer program product executes, executes the large model-based target positioning method proposed in the above embodiment of the present application.

[0161] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0162] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A target positioning method based on a large model, comprising: Receiving a positioning request sent by a target terminal; wherein the positioning request includes a target image and a voice command; Extracting first object information of the object to be located from the voice command, and performing object detection on the target image according to the first object information to obtain a detection result; wherein the detection result includes position information of the candidate object in the target image; According to the position information of the candidate object in the target image, intercepting the object image of the candidate object from the target image; Determine the target object from the candidate objects using a large model according to the object picture, the position information of the candidate object in the target picture and the first object information; The position information of the target object in the target picture is sent to the terminal, so that the target terminal determines the position information of the target object relative to the target terminal according to the position information of the target object in the target picture.

2. The method of claim 1, wherein: The determining the target object from the candidate objects using a large model according to the object picture, the position information of the candidate object in the target picture and the first object information includes: Acquire prompt information according to the object picture, the position information of the candidate object in the target picture and the first object information; Inputting the prompt information into the large model to determine whether the candidate object meets the positioning requirements; The target object is determined according to the judgment result of the large model.

3. The method of claim 2, wherein: The detection result also includes a detection score, and the target object is determined according to the judgment result of the large model, including: In response to determining that there are multiple candidate objects that meet the positioning requirement according to the judgment result, a candidate object with the highest detection score among the multiple candidate objects is determined as the target object.

4. The method of claim 2, wherein: The step of inputting the first prompt information into the large model to determine whether the candidate object meets the positioning requirement includes: Using the large model, extracting object information from the object image and the position information of the candidate object in the target image to obtain second object information; According to the matching condition between the second object information and the first object information, it is determined whether the candidate object meets the positioning requirement.

5. The method of claim 1, wherein: The extracting first object information of the object to be located from the voice command includes: Performing voice recognition on the voice command to convert the voice command into text; The large model is used to extract object information from the text to obtain the first object information.

6. The method of claim 1, wherein: The performing target detection on the target image according to the first object information to obtain a detection result includes: Performing feature extraction on the first object information to obtain a first text feature; Performing feature extraction on the target image to obtain a first image feature; fusing the first text feature and the first image feature to obtain a second text feature and a second image feature; Determining a target image feature having the highest matching degree with the second text feature among the second image features; Decoding is performed according to the target image features to obtain the detection result.

7. The method of claim 1, wherein: The target terminal includes a distance measuring device, and the target terminal determines the position information of the target object relative to the target terminal according to the position information of the target object in the target picture, including: The target terminal measures the distance between the target object and the target terminal by using the distance measuring device according to the position information of the target object in the target picture; According to the distance, position information of the target object relative to the target terminal is determined.

8. The method according to any one of claims 1 to 7, wherein: The target terminal is a robot having a mechanical arm. The target terminal is also used to control the mechanical arm to pick up the target object according to position information of the target object relative to the target terminal.

9. A target positioning device based on a large model, comprising: A receiving module, configured to receive a positioning request sent by a target terminal; wherein the positioning request includes a target image and a voice command; An extraction module, used to extract first object information of the object to be located from the voice command; A detection module, configured to perform target detection on the target image according to the first object information to obtain a detection result; wherein the detection result includes position information of the candidate object in the target image; A screenshot module, configured to capture an object image of the candidate object from the target image according to position information of the candidate object in the target image; A retrieval module, configured to determine a target object from the candidate objects using a large model according to the object image, the position information of the candidate object in the target image and the first object information; The sending module is used to send the position information of the target object in the target picture to the terminal, so that the target terminal determines the position information of the target object relative to the target terminal according to the position information of the target object in the target picture.

10. The device of claim 9, wherein: The retrieval module is used to: Acquire prompt information according to the object picture, the position information of the candidate object in the target picture and the first object information; Inputting the prompt information into the large model to determine whether the candidate object meets the positioning requirements; The target object is determined according to the judgment result of the large model.

11. The device of claim 10, wherein: The detection result also includes a detection score, and the retrieval module is used to: In response to determining that there are multiple candidate objects that meet the positioning requirement according to the judgment result, a candidate object with the highest detection score among the multiple candidate objects is determined as the target object.

12. The device of claim 10, wherein: The retrieval module is used to: Using the large model, extracting object information from the object image and the position information of the candidate object in the target image to obtain second object information; According to the matching condition between the second object information and the first object information, it is determined whether the candidate object meets the positioning requirement.

13. The device of claim 9, wherein: The extraction module is used to: Performing voice recognition on the voice command to convert the voice command into text; The large model is used to extract object information from the text to obtain the first object information.

14. The device of claim 9, wherein: The detection module is used for: Performing feature extraction on the first object information to obtain a first text feature; Performing feature extraction on the target image to obtain a first image feature; fusing the first text feature and the first image feature to obtain a second text feature and a second image feature; Determining a target image feature having the highest matching degree with the second text feature among the second image features; Decoding is performed according to the target image features to obtain the detection result.

15. The device of claim 9, wherein: The target terminal includes a distance measuring device, and the target terminal measures the distance between the target object and the target terminal by using the distance measuring device according to the position information of the target object in the target picture; According to the distance, position information of the target object relative to the target terminal is determined.

16. The device according to any one of claims 9 to 15, wherein: The target terminal is a robot having a mechanical arm. The target terminal is also used to control the mechanical arm to pick up the target object according to position information of the target object relative to the target terminal.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images