Shop retrieval method and device based on large model technology, equipment and medium

Abstract

The invention provides a store retrieval method based on a large model technology, and relates to the technical field of artificial intelligence, in particular to the technical field of natural language processing, computer vision, large models and generative models. The specific implementation scheme is as follows: according to received input information, generating a store description text of a to-be-determined store; determining respective sub-images of a plurality of shops to be matched from environment images collected by the moving object in the moving process; and determining a target shop from the plurality of shops to be matched according to the shop description text and the respective sub-images of the plurality of shops to be matched. The invention further provides a store retrieval device based on the large model technology, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of natural language processing, computer vision, large models, and generative model technology. More specifically, the present disclosure provides a store search method, device, electronic device, storage medium, and computer program product based on large model technology. Background Art

[0002] Autonomous driving technology is now available. Although the efficiency and benefits of machines are far better than those of real people, there are some inconvenient scenarios without real drivers. For example, when passengers want to find nearby stores, real drivers can quickly and accurately inform passengers because they are familiar with the surrounding conditions. However, autonomous driving programs are not so smart at present. First, they cannot communicate with passengers through natural language to obtain information, and second, they cannot use the information obtained to search for stores. Therefore, it is difficult to meet user needs. Summary of the invention

[0003] The present disclosure provides a store search method, device, electronic device, storage medium and computer program product based on big model technology.

[0004] According to the first aspect, a store retrieval method based on a large model technology is provided, the method comprising: generating a store description text of a store to be determined based on received input information; determining a plurality of sub-images of stores to be matched from environmental images collected by a mobile object during movement; and determining a target store from a plurality of stores to be matched based on the store description text and the sub-images of the stores to be matched.

[0005] According to the second aspect, a store retrieval device based on large model technology is provided, which includes: a description text generation module, used to generate a store description text of a store to be determined according to received input information; an image determination module, used to determine a plurality of sub-images of stores to be matched from environmental images collected by a mobile object during movement; and a target store determination module, used to determine a target store from a plurality of stores to be matched according to the store description text and the sub-images of the plurality of stores to be matched.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an exemplary system architecture to which a store search method and device based on a large model technology can be applied according to an embodiment of the present disclosure;

[0012] Figure 2 is a flow chart of a store search method based on a large model technology according to an embodiment of the present disclosure;

[0013] Figure 3 is a flow chart of a store search method based on a large model technology according to another embodiment of the present disclosure;

[0014] Figure 4 is a service call diagram of a serverless architecture according to an embodiment of the present disclosure;

[0015] Figure 5 is a block diagram of a store search device based on a large model technology according to an embodiment of the present disclosure; and

[0016] Figure 6 The present invention is a block diagram of an electronic device for implementing a store search method based on a large model technology according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0019] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0020] When a user is riding in an autonomous driving vehicle and wants to find stores nearby, existing autonomous driving programs cannot communicate with the user in natural language or perform searches, and therefore cannot meet the user's needs.

[0021] In the robot delivery scenario, after planning the robot's delivery route, if there are surrounding stores as a reference, the robot can be assisted in quickly determining the location and direction, thereby improving the robot's delivery efficiency.

[0022] The embodiment of the present disclosure provides a store search method based on big model technology, through which the big model can directly communicate with the user through natural language and obtain search information, and the big model has image understanding ability, and in the autonomous driving scenario, it can determine whether the image collected by the vehicle contains the store the user wants to find. In the robot delivery scenario, by determining the stores around the robot, the robot can be assisted to quickly determine the accuracy of the delivery path.

[0023] The demand for determining stores mentioned above may have obvious peaks and troughs. For example, the demand for users to find stores by autonomous driving vehicles is higher in the evening and weekends (for example, there are many people eating and shopping), but lower in the daytime and at night. The demand for robot delivery may arise during user-specific holidays or specific time periods. Therefore, the trough time of the demand for determining stores mentioned above may be longer, and deploying dedicated servers may result in a waste of resources.

[0024] In view of this, the embodiments of the present disclosure adopt the Serverless system architecture to achieve the above-mentioned needs of users. Serverless is serverless computing, which provides a micro architecture. End customers do not need to deploy, configure or manage server services. The servers required for code operation are all provided by the cloud platform.

[0025] The serverless system architecture of the embodiment of the present disclosure refers to the use of at least part of the services of the store search method based on the big model technology of the embodiment of the present disclosure without setting up a dedicated server, and at least part of the services can be implemented by the cloud platform. During the peak phase, multiple servers of the cloud platform can be opened to implement at least part of the services. During the trough phase, a small number of servers of the cloud platform can be opened to implement at least part of the services.

[0026] Figure 1 is a schematic diagram of an exemplary system architecture of a store search method and device based on a large model technology that can be applied according to an embodiment of the present disclosure. It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0027] like Figure 1 As shown, the system architecture 100 according to this embodiment may include an autonomous driving vehicle 101, a robot 102, a network 103, and servers 104, 105, and 106. The network 103 is used to provide a medium for communication links between the autonomous driving vehicle 101, the robot 102, and the servers 104, 105, and 106. The network 103 may include various connection types, such as wired and / or wireless communication links, and the like.

[0028] For example, a terminal with a display screen may be provided inside the autonomous driving vehicle 101. When the user rides the autonomous driving vehicle 101, the user may interact with the terminal, for example, by inputting voice, text, or uploading images via Bluetooth. The autonomous driving vehicle 101 may also be provided with a camera, for example, a smart camera is provided on the top of the vehicle, and the smart camera may be turned on when the user needs to search for nearby stores and collect images around the vehicle. The terminal device may call the big model based on the user's input information and the images collected by the vehicle to determine whether there are stores that the user wants to find around the vehicle. The big model service may be provided with a corresponding dedicated service to implement. For example, the server 104 provides a big model service. Before inputting data into the big model, the input data generally needs to be preprocessed, for example, converting the user's voice into text, identifying the stores to be matched from the images collected by the vehicle, etc. These services may be implemented by the server of the cloud platform in the Serverless mode. For example, servers 105 and 106 are servers of the cloud platform, which may be used to implement services such as voice recognition and identification of stores to be matched in images.

[0029] For another example, the robot 102 is a delivery robot. After the robot 102 plans the delivery route, auxiliary positioning information on the route can also be provided to the robot 102, such as there is an XX store at a certain intersection, and there is a YY store on the right side of the road. The robot 102 can collect surrounding images during movement, and call the large model to determine the position and direction of the robot 102 on the road based on the auxiliary positioning information and the collected images, so that the robot 102 can quickly determine the accuracy of the delivery path.

[0030] The embodiments of the present disclosure can improve the store recognition capability and meet the user's store search needs by calling the big model to determine the store information around the autonomous driving vehicle during driving. By calling the big model to determine the store information around the robot during movement, the robot can be assisted in determining the accuracy of the delivery path.

[0031] Figure 2 It is a flowchart of a store search method based on big model technology according to an embodiment of the present disclosure.

[0032] like Figure 2 As shown, the store search method 200 based on the big model technology includes operations S210 to S230.

[0033] In operation S210, a store description text of the store to be determined is generated according to the received input information.

[0034] Users can input voice, text or upload images via Bluetooth to the terminal set up in the autonomous vehicle. Or users can input voice, text or upload images to the display screen of the delivery robot.

[0035] For input voice and input text, the description information of the store to be determined can be extracted, such as the store name, type, etc. The type can include whether the store is a restaurant or a clothing store. For uploaded images, the color of the store, the text on the sign, and other information can be extracted from the image to generate a description text.

[0036] The description information extracted from the input voice and the input text and the description information extracted from the input image may be combined to obtain a store description text of the store to be determined.

[0037] In operation S220, a plurality of sub-images of stores to be matched are determined from the environment images collected by the mobile object during its movement.

[0038] The moving object can be an autonomous vehicle or a robot.

[0039] In the scenario of autonomous driving, when it is determined that the user has the need to find surrounding stores, such as when the user inputs voice, text or images containing store information, the camera installed on the vehicle can be turned on to collect the environmental image around the vehicle, and the corresponding image processing service can be called to extract the sub-image containing the store from the environmental image. The camera on the autonomous driving vehicle can be a smart camera. During the image collection process, the orientation information of the store in the collected environmental image can be determined based on the acquisition angle and other information. The orientation information can be an absolute orientation such as east, south, west, north, etc. It can also be a relative orientation, such as the left front and right front relative to the vehicle.

[0040] In the robot delivery scenario, the robot can collect images of the surrounding environment while moving, and call the corresponding image processing service to extract the sub-image containing the store from the environment image. The robot's camera can also be a smart camera, and during the acquisition process, the orientation information of the store in the acquired image can also be determined according to the acquisition angle. The orientation information can be an absolute orientation such as east, south, west, north, or south. It can also be a relative orientation, such as the left front or right front relative to the vehicle.

[0041] The store in the environment image is the store to be matched. The image processing service can identify the store and the store location in the environment image by performing target detection on the environment image, and extract a screenshot containing the store from the environment image according to the store location as the sub-image of the store to be matched. Since each environment image has its own orientation information, each sub-image containing a store extracted from the environment image can contain orientation information.

[0042] In operation S230 , a target store is determined from the multiple stores to be matched according to the store description text and the sub-images of the multiple stores to be matched.

[0043] For example, a multimodal big model can be called to input the store description text and the sub-images of each store to be matched into the big model. The big model processes the store description text and each sub-image, and determines the sub-image that matches the store description text from multiple sub-images. The store to be matched in the sub-image is the target store.

[0044] The large model can return a sub-image of the target store to the terminal in the autonomous driving vehicle, and the terminal can display the sub-image of the target store and the location information of the sub-image, so that the user can quickly confirm the store to be found.

[0045] The large model can also return a sub-image of the target store to the robot, which can then use the location information in the sub-image to help confirm its own position and direction, thereby helping to confirm the accuracy of the delivery route.

[0046] The embodiments of the present disclosure generate a store description text of the store to be determined according to the user's input information, capture a plurality of sub-images of the stores to be determined from the environmental images collected by the mobile object during its movement, and determine the target store from the stores to be determined based on the store description text and the sub-images of the stores to be determined. This can accurately understand user needs, improve the accuracy of store recognition, meet user needs, and enhance user experience.

[0047] In the autonomous driving scenario, it can meet the needs of users to find nearby stores and improve the user experience. In the robot delivery scenario, it can assist the robot in determining the accuracy of the delivery route and improve the delivery efficiency.

[0048] Figure 3 It is a flowchart of a store search method based on big model technology according to another embodiment of the present disclosure.

[0049] like Figure 3 As shown, this embodiment includes operations S311 to S314, operations S321 to S323, and operations S331 to S332. Operations S311 to S314 are steps for processing user input information. Operations S321 to S323 are steps for processing images collected from moving objects. Operations S331 to S332 are steps for calling large model services for processing.

[0050] In operation S311, input information is received, the input information including at least one of an input voice, an input text, and an input image.

[0051] In operation 312, the input speech is converted into a converted text, and a first feature description text is extracted from at least one of the converted text and the input text.

[0052] According to an embodiment of the present disclosure, a description text including at least one of the name, type, and logo of the store to be determined is extracted from at least one of the converted text and the input text as the first feature description text.

[0053] For example, the converted text obtained by converting the input speech can be combined with the input text to obtain a combined text. Next, the large language model can be called to summarize and refine the combined text, and the name, type, logo, color, shape, and surrounding objects of the store that the user wants to find can be extracted from the combined text. Types include, for example, restaurants, clothing stores, shoe stores, etc., and logos can include store icons, image logos, etc. The description text of these features is extracted from the combined text as the first feature description text.

[0054] In operation 313 , a second feature description text of the store to be determined is extracted from the input image.

[0055] According to an embodiment of the present disclosure, at least one of shape, color and text is extracted from an input image; and a description text of at least one of the shape, color and text is generated as a second feature description text.

[0056] For example, a multimodal large model can be called to extract the shape, color, text, etc. of the store from the input image, describe the shape and color with text, and then combine the shape and color description text with the text in the image to obtain a second feature description text.

[0057] In operation S314, a store description text is determined according to at least one of the first feature description text and the second feature description text.

[0058] For example, the first feature description text and the second feature description text may be integrated, and the same features may be merged to obtain a store description text.

[0059] In operation S321, an environment image collected by the moving object during its movement is acquired.

[0060] In operation S322 , a plurality of stores to be matched and the position of each store to be matched are identified from the environment image.

[0061] In operation S323, a plurality of sub-images of the stores to be matched are captured from the environment image.

[0062] For example, an autonomous vehicle collects images of the surrounding environment while moving, or a mobile robot collects images of the surrounding environment while moving.

[0063] For example, the image processing service may be called to identify the stores to be matched from the environment image, and the location of each store to be matched may be marked, and a sub-image of each store to be matched may be cut out from the environment image according to the location.

[0064] For example, based on the angle of the camera when capturing each environmental image, the orientation information of each environmental image can be determined, thereby determining the orientation information of each sub-image of the store to be matched. The orientation information can represent the absolute orientation of the store to be matched in the sub-image, such as east, south, west or north, or it can also represent the relative orientation of the store to be matched, such as the orientation relative to the vehicle or robot, such as in the left front or right front of the vehicle / robot.

[0065] In operation S331 , a target store is determined from a plurality of stores to be matched according to the store description text and the sub-images of the plurality of stores to be matched.

[0066] For example, the multimodal big model can be called, and the store description text and the sub-images of multiple stores to be matched can be input into the multimodal big model. The multimodal big model processes the store description text and each sub-image, and determines the sub-image that matches the store description text from multiple sub-images. The store in the sub-image is the target store.

[0067] In operation S332, the sub-image of the target store and the location information of the target store are output.

[0068] For example, the multimodal large model can return the matched sub-image of the target store to the terminal of the autonomous driving vehicle, and the terminal can display the sub-image of the target store and the orientation information of the sub-image. The large model can also return multiple target stores, which can be displayed on the display screen of the terminal in the form of a list, and the user can confirm the final target store.

[0069] For example, the multimodal large model can return the matched sub-image of the target store to the robot, and the robot can assist in determining the accuracy of its own position based on the orientation information of the sub-image of the target store.

[0070] The embodiments of the present disclosure call a large model to extract a first feature description text from an input voice and an input text and a second feature description text from an input image, generate a store description text for a store to be determined based on the first feature description text and the second feature description text, capture a plurality of sub-images of stores to be matched from an environmental image collected during the movement of the mobile object, and then call the large model to determine a target store from the stores to be matched based on the store description text and the sub-images of the stores to be matched, thereby improving the accuracy of store recognition.

[0071] Figure 4 It is a service call diagram of a serverless architecture according to an embodiment of the present disclosure.

[0072] The embodiment of the present disclosure can use the Serverless architecture to implement a store search method based on the big model technology. Figure 4 As shown, the Serverless architecture provides an access service 40, which is used to call other services to process user requests. Other services include a speech conversion service 41, a text feature extraction service 42, an image feature extraction service 43, a search service 44, and an image processing service 45. The image captured by the camera is also transmitted to the access service 40, and the access service 40 can call the image processing service 45 to identify the store to be matched based on the image transmitted by the camera.

[0073] For example, the terminal is a terminal in an autonomous driving vehicle or a robot terminal. The user inputs voice, text or image through the terminal, and then the terminal can transmit the voice, text or image input by the user to the access service 40 by initiating an http request. For the input voice, the access service 40 can call the voice conversion service 41 to convert the input voice into converted text. For the converted text and the text input by the user, the access service 40 can call the text feature extraction service 42 for processing, and the text feature extraction service 42 can specifically call the large model service 431 to extract features for the converted text and the input text to obtain a first feature description text. For example, the large model service 431 can include a large language model, and the large model service 431 can call the large language model to summarize the converted text and the input text to obtain a first feature description text. For the input image, the access service 40 can call the image feature extraction service 43 for processing, and the image feature extraction service 43 can specifically call the large model service 431 to process the input image to obtain a second feature description text. For example, the big model service 431 may include a multimodal big model, and the big model service 431 may call the multimodal big model to extract features from the input image and generate a description text of the extracted image features, namely, a second feature description text.

[0074] For example, the camera may be a camera installed on an autonomous vehicle or a robot. During the movement of the autonomous vehicle or the robot, the camera collects images of the surrounding environment and transmits the collected images to the access service 40 by initiating an http request. The access service 40 calls the image processing service 45 to process the environment image. The image processing service 45 may specifically call the target detection service 451 for processing. For example, the target detection service 451 may use a target detection model to identify the store to be matched from the environment image and extract a sub-image of each store to be matched from the environment image. The target detection model may be, for example, a YOLO model. The image processing service 45 may also call the cache service 452 to cache the sub-images of each store to be matched extracted by the target detection service 451.

[0075] The retrieval service 44 can obtain the sub-images of each store to be matched from the cache service 452, and then call the large model service 431 to determine the target store from multiple stores to be matched according to the store description text and the sub-images of each store to be matched.

[0076] The search service may return the sub-image and location information of the target store to the access service 40, and the access service 40 may return it to the terminal. The terminal may display the sub-image and location information of the target store to the user, so that the user can quickly determine the store to be found.

[0077] The embodiments of the present disclosure adopt a Serverless architecture, which does not require the setting of a dedicated server. By transmitting user input information and images captured by the camera to the access service, the access service calls various services to implement a store retrieval method based on large model technology, which can reduce resource usage and save costs.

[0078] According to an embodiment of the present disclosure, the present disclosure also provides a store search device based on large model technology.

[0079] Figure 5 It is a block diagram of a store search device based on a large model technology according to an embodiment of the present disclosure.

[0080] like Figure 5 As shown, the store search device 500 based on the large model technology includes a description text generation module 510, an image determination module 520 and a target store determination module 530.

[0081] The description text generation module 510 is used to generate a store description text of the store to be determined according to the received input information.

[0082] The image determination module 520 is used to determine the respective sub-images of a plurality of stores to be matched from the environment images collected by the mobile object during its movement.

[0083] The target store determination module 530 is used to determine a target store from a plurality of stores to be matched according to the store description text and the respective sub-images of the plurality of stores to be matched.

[0084] According to an embodiment of the present disclosure, the input information includes at least one of input voice, input text and input image. The description text generation module 510 includes a first feature description text generation unit, a second feature description text generation unit and a store description text generation unit.

[0085] The first feature description text generating unit is used to extract a first feature description text of the store to be determined from at least one of the input voice and the input text.

[0086] The second feature description text generating unit is used to extract the second feature description text of the store to be determined from the input image.

[0087] The store description text generating unit is used to determine the store description text according to at least one of the first feature description text and the second feature description text.

[0088] The first feature description text generating unit includes a conversion subunit and a first extraction subunit.

[0089] The conversion subunit is used to convert the input speech into converted text.

[0090] The first extraction subunit is used to extract a description text containing at least one of the name, type and logo of the store to be determined from at least one of the conversion text and the input text as the first feature description text.

[0091] The second feature description text generating unit includes a second extracting subunit and a generating subunit.

[0092] The second extraction subunit is used to extract at least one of shape, color and text from the input image.

[0093] The generating subunit is used to generate a description text of at least one of shape, color and text as the second feature description text.

[0094] The image determination module 520 includes an acquisition unit, a recognition unit, and a capture unit.

[0095] The acquisition unit is used to acquire the environment image collected by the moving object during its movement.

[0096] The recognition unit is used to recognize multiple stores to be matched from the environment image.

[0097] The capture unit is used to capture the sub-images of the multiple stores to be matched from the environment image.

[0098] The store search device 500 based on the large model technology also includes a location information determination module and an output module.

[0099] The position information determination module is used to determine the position information of each of the multiple stores to be matched in the environment image.

[0100] The output module is used to output the sub-image of the target store and the location information of the target store.

[0101] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0102] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0103] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0104] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0105] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a store retrieval method based on a large model technology. For example, in some embodiments, a store retrieval method based on a large model technology may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the store retrieval method based on the large model technology described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the store search method based on the large model technology in any other appropriate manner (for example, by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0108] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0111] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A store search method based on a large model technology, comprising: Generate a store description text of the store to be determined based on the received input information; Determine, from the environment images collected by the mobile object during its movement, respective sub-images of a plurality of stores to be matched; as well as A target store is determined from the multiple stores to be matched according to the store description text and the sub-images of the multiple stores to be matched.

2. The method according to claim 1, wherein: The input information includes at least one of input voice, input text and input image; and generating a store description text of the store to be determined according to the received input information includes: extracting a first feature description text of the to-be-determined store from at least one of the input voice and the input text; Extracting a second feature description text of the store to be determined from the input image; and The store description text is determined according to at least one of the first feature description text and the second feature description text.

3. The method according to claim 2, wherein: The extracting the first feature description text of the to-be-determined store from at least one of the input voice and the input text comprises: Converting the input speech into converted text; and A description text including at least one of the name, type and logo of the store to be determined is extracted from at least one of the converted text and the input text as the first feature description text.

4. The method according to claim 2, wherein: The step of extracting the second feature description text of the to-be-determined store from the input image comprises: Extracting at least one of shape, color and text from the input image; and Generate a description text of at least one of the shape, color and text as the second feature description text.

5. The method according to claim 1, wherein: The step of determining the respective sub-images of a plurality of stores to be matched from the environment image collected by the mobile object during its movement comprises: Acquire an environmental image collected by the moving object during its movement; Identifying the plurality of stores to be matched from the environment image; and The sub-images of the plurality of stores to be matched are captured from the environment image.

6. The method according to claim 1, further comprising: Determine the position information of each of the plurality of stores to be matched in the environment image; as well as The sub-image of the target store and the location information of the target store are output.

7. A store search device based on large model technology, comprising: A description text generation module, used to generate a store description text of the store to be determined according to the received input information; An image determination module, used to determine the sub-images of multiple stores to be matched from the environment images collected by the mobile object during its movement; as well as The target store determination module is used to determine a target store from the multiple stores to be matched according to the store description text and the respective sub-images of the multiple stores to be matched.

8. The device according to claim 7, wherein: The input information includes at least one of input speech, input text and input image; the description text generation module includes: A first feature description text generating unit, configured to extract a first feature description text of the to-be-determined store from at least one of the input speech and the input text; A second feature description text generating unit is used to extract a second feature description text of the store to be determined from the input image; and A store description text generating unit is used to determine the store description text according to at least one of the first feature description text and the second feature description text.

9. The device according to claim 8, wherein: The first feature description text generation unit includes: a conversion subunit, configured to convert the input speech into a conversion text; and The first extraction subunit is used to extract a description text containing at least one of the name, type and logo of the store to be determined from at least one of the conversion text and the input text as the first feature description text.

10. The device according to claim 8, wherein: The second feature description text generating unit includes: A second extraction subunit, configured to extract at least one of shape, color and text from the input image; and The generating subunit is used to generate a description text of at least one of the shape, color and text as the second feature description text.

11. The device according to claim 7, wherein: The image determination module comprises: An acquisition unit, used to acquire an environment image collected by the moving object during its movement; an identification unit, configured to identify the plurality of stores to be matched from the environment image; and The capturing unit is used to capture the respective sub-images of the multiple stores to be matched from the environment image.

12. The apparatus according to claim 7, further comprising: A position information determination module, used to determine the position information of each of the multiple stores to be matched in the environment image; as well as An output module is used to output the sub-image of the target store and the location information of the target store.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images