Service providing method, system, device and equipment for unmanned store and medium

Abstract

The embodiment of the invention provides a service providing method, system, device and equipment for an unmanned store, and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: inputting multi-modal data and event information of a to-be-processed service event into a preset large model when the to-be-processed service event triggered by a target customer in a target store is detected, obtaining to-be-replied response information output by the preset large model; the multi-modal data comprises multiple items of the following items: a user tag of a target customer, weather data, actual data, hotspot data, audio data and video data of the target customer, location information, name and type of a target store, commodities, attribute information of a shelf, historical order information of the target customer, and historical service data for the target customer; and generating a service voice corresponding to the to-be-replied response information, and sending the service voice to a target terminal in the target store, so that the target terminal plays the service voice through the digital person. And the shopping experience of customers can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, system, device, equipment and medium for providing services in an unmanned store. Background Art

[0002] With the development of technology, the retail industry has begun to see the emergence of cloud-attended business models. Under the cloud-attended business model, there are no staff on site to provide shopping guides, checkout and other services, which often requires customers to spend time and energy looking for goods and figuring out settlement operations, resulting in a poor shopping experience for customers. Therefore, how to improve the shopping experience of customers in stores under the cloud-attended model (which can be called unmanned stores) has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method, system, device, equipment and medium for providing services in an unmanned store, so as to improve the shopping experience of customers in the unmanned store. The specific technical solution is as follows:

[0004] In a first aspect of an embodiment of the present application, a service providing method of an unmanned store is provided, the method comprising:

[0005] When a service event to be processed is detected, which is triggered by a target customer in a target store, the multimodal data to be used and the event information of the service event to be processed are input into the current preset macro model, and the response information of the service indicated by the service event to be processed output by the current preset macro model is obtained as the response information to be replied;

[0006] The multimodal data to be used currently includes multiple items of the following: user tags of the target customer, current weather data, current real-time data, current hot spot data, currently collected audio data of the target customer, currently collected video data of the target customer, location information of the target store, name and type of the target store, attribute information of goods in the target store, attribute information of shelves in the target store, historical order information of the target customer, and historical service data for the target customer;

[0007] A voice corresponding to the response information to be replied is generated as a service voice, and the service voice is sent to a target terminal in the target store, so that the target terminal plays the service voice through a digital human.

[0008] Optionally, the method is applied to a server, the server comprising: a data processing end, an inference end and a speech synthesis end; the inference end is deployed with a current preset large model; the target store is provided with a collection device for collecting video data and / or audio data in the target store;

[0009] When a service event to be processed is detected to be triggered by a target customer in a target store, the method further includes: inputting the multimodal data to be used and the event information of the service event to be processed into the current preset macro model, and obtaining the response information of the service indicated by the service event to be processed output by the current preset macro model as the response information to be replied.

[0010] The data processing end acquires the currently acquired data from the acquisition device, performs behavior recognition on the acquired data, determines the event information of the service event to be processed triggered by the target customer, and sends the determined event information of the service event to be processed to the reasoning end;

[0011] When a service event to be processed is detected and triggered by a target customer in a target store, the multimodal data to be used and the event information of the service event to be processed are input into the current preset macro model, and response information for the service indicated by the service event to be processed output by the current preset macro model is obtained as the response information to be replied, including:

[0012] When receiving the event information of the pending service event sent by the data processing end, the reasoning end inputs the multimodal data to be used and the event information of the pending service event into the current preset large model, obtains the response information of the service indicated by the pending service event output by the current preset large model as the pending reply response information, and sends the pending reply response information to the speech synthesis end;

[0013] The generating of the voice corresponding to the response information to be replied as the service voice includes:

[0014] The speech synthesis end generates speech corresponding to the received response information to be replied as a service speech.

[0015] Optionally, when the target customer is identified as entering the target store from the video data acquired by the acquisition device, the event information of the service event to be processed determined represents an entry event, and the service voice represents a welcome message;

[0016] In the case where the audio data acquired from the acquisition device represents the shopping guide service request of the target customer, the event information of the service event to be processed determined represents a shopping guide event, and the service voice represents a service response to the shopping guide service request.

[0017] Optionally, when receiving the event information of the service event to be processed sent by the data processing end, the reasoning end inputs the multimodal data to be currently utilized and the event information of the service event to be processed into the current preset large model, and obtains the response information of the service indicated by the service event to be processed output by the current preset large model as the response information to be replied, the method further includes:

[0018] When the target customer is identified as entering the target store from the video data acquired from the acquisition device, the data processing end performs feature recognition on the video data acquired from the acquisition device to obtain a user tag of the target customer, and sends the obtained user tag to the inference end;

[0019] and / or,

[0020] The inference end obtains the customer identifier carried in the door opening request for the target store triggered by the target customer using the user terminal, and obtains the user tag corresponding to the obtained customer identifier from a preset database; wherein the user tags in the preset database include: tags obtained based on registration information when the customer represented by each customer identifier registers using the user terminal, and / or tags obtained based on historical purchase records of each customer identifier.

[0021] Optionally, sending the determined event information of the service event to be processed to the inference end includes:

[0022] In the case where the event information of the determined service event to be processed indicates a shopping guide event and the condition for real person taking over is not met, a first message containing the event information of the determined service event to be processed is sent to the on-duty terminal used by the staff of the target store, so that the on-duty terminal displays the event information of the determined service event to be processed, and sends the event information of the determined service event to be processed to the reasoning end; wherein the real person taking over condition includes at least one of the following: the number of dialogue rounds between the digital human and the target customer is greater than a preset number, the data processing end detects that the target customer has negative emotions based on the data obtained from the acquisition device, and the service information of the service indicated by the determined service event to be processed does not exist in the preset knowledge base;

[0023] The method further comprises:

[0024] After generating the pending reply response information for the service indicated by the pending service event, the reasoning end sends the pending reply response information to the on-duty terminal so that the on-duty terminal displays the pending reply response information;

[0025] Before the speech synthesis terminal generates the speech corresponding to the received response information to be replied as the service speech, the method further includes:

[0026] When the event information of the determined service event to be processed indicates a shopping guide event and the conditions for real-person takeover are met, the data processing end sends a second message containing the event information of the determined service event to be processed to the on-duty terminal, so that the on-duty terminal displays the event information of the determined service event to be processed, and when the staff inputs the pending reply information for the service indicated by the pending service event in the on-duty terminal based on the event information and reply information of the displayed service event, the pending reply information is sent to the speech synthesis end.

[0027] Optionally, the method further includes:

[0028] For each service event triggered by the target customer, the data processing end sends the associated information of the service event to the optimization processing end, so that the optimization processing end updates the recognition model currently deployed in the data processing end for recognition based on the received associated information; wherein the associated information of any service event includes: event information of the service event, and data collected by the collection device when the target customer triggers the service event;

[0029] and / or,

[0030] For each service event triggered by the target customer, the inference end sends the input data and output data of the current preset large model in the process of processing the service event to the optimization processing end, so that the optimization processing end updates the current preset large model based on the data received from the inference end.

[0031] Optionally, the speech synthesis terminal generates a speech corresponding to the received response information to be replied as a service speech, including:

[0032] The speech synthesis end generates, based on a voice cloning algorithm, a speech corresponding to the response information to be replied represented by a specified timbre as a service speech.

[0033] Optionally, before inputting the multimodal data to be currently utilized and the event information of the service event to be processed into the current preset macro model, and obtaining the response information of the service indicated by the service event to be processed output by the current preset macro model as the response information to be replied, the method further includes:

[0034] The reasoning end searches for response information for the service indicated by the to-be-processed service event from a preset knowledge base to obtain standard response information;

[0035] The step of inputting the multimodal data to be currently utilized and the event information of the service event to be processed into the current preset macro model, and obtaining the response information of the service indicated by the service event to be processed output by the current preset macro model as the response information to be replied includes:

[0036] The standard response information, the event information of the service event to be processed, and the multimodal data to be used are input into the current preset large model, and combined with the random factor, the response information of the service indicated by the service event to be processed output by the current preset large model is obtained as the response information to be replied.

[0037] According to a second aspect of the embodiments of the present application, there is further provided a service providing system for an unmanned store, the system comprising: a service end and a target terminal in a target store;

[0038] The server is used to execute any of the above-mentioned unmanned store service provision methods;

[0039] The target terminal is used to receive the service voice sent by the service end and play the service voice through the digital human.

[0040] Optionally, the system further comprises: an on-duty terminal used by staff of the target store;

[0041] The on-duty terminal is used to, when receiving a first message containing event information of a determined service event to be processed sent by the data processing end, display the event information of the determined service event to be processed, and send the event information of the determined service event to be processed to the reasoning end; when receiving response information to be replied sent by the reasoning end, display the received response information to be replied;

[0042] When a second message containing event information of a determined service event to be processed is received from the data processing end, the event information of the determined service event to be processed is displayed, and when the staff inputs the pending reply information for the service indicated by the pending service event in the on-duty terminal based on the event information and reply information of the displayed service event, the pending reply information is sent to the speech synthesis end.

[0043] Optionally, the system further includes: an optimization processing end;

[0044] The target terminal is further used to send the voice played by the digital human in the process of processing each service event triggered by the target customer to the optimization processing end;

[0045] The on-duty terminal is further used to send the interface content displayed by itself during the process of processing each service event triggered by the target customer to the optimization processing end;

[0046] The optimization processing end is used to update the recognition model currently deployed in the data processing end for recognition based on the associated information of each service event received from the data processing end, and / or to update the current preset large model based on the input data and output data of the current preset large model in the process of processing each service event received from the inference end.

[0047] Optionally, the system further includes: a collection device provided in the target store;

[0048] The acquisition device is used to acquire video data and / or audio data in the target store.

[0049] Optionally, the system further includes: a merchant management terminal;

[0050] The merchant management end is used to record the attribute information of the goods in the target store, the attribute information of the shelves in the target store, and the historical order information of the customers of the target store, and synchronize the recorded information to the inference end.

[0051] Optionally, the target terminal is specifically configured to drive and render the digital human according to a preset digital human model and the service voice when receiving the service voice, so that the lip movements of the displayed digital human match the service voice.

[0052] According to a third aspect of the embodiments of the present application, there is further provided a service providing device for an unmanned store, the device comprising:

[0053] A response information determination module is used to input the current multimodal data to be utilized and the event information of the service event to be processed into the current preset macro model when a service event to be processed is detected to be triggered by a target customer in a target store, and obtain the response information of the service indicated by the service event to be processed output by the current preset macro model as the response information to be replied;

[0054] The multimodal data to be used currently includes multiple items of the following: user tags of the target customer, current weather data, current real-time data, current hot spot data, currently collected audio data of the target customer, currently collected video data of the target customer, location information of the target store, name and type of the target store, attribute information of goods in the target store, attribute information of shelves in the target store, historical order information of the target customer, and historical service data for the target customer;

[0055] The service voice generation module is used to generate the voice corresponding to the response information to be replied as the service voice, and send the service voice to the target terminal in the target store so that the target terminal plays the service voice through the digital human.

[0056] The present application also provides an electronic device, including:

[0057] Memory, used to store computer programs;

[0058] The processor is used to implement any of the unmanned store service provision methods described above when executing the program stored in the memory.

[0059] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the unmanned store service providing method described above is implemented.

[0060] An embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute any of the unmanned store service providing methods described above.

[0061] Beneficial effects of the embodiments of the present application:

[0062] A service providing method for an unmanned store provided by an embodiment of the present application can, when a pending service event triggered by a target customer in a target store is detected, input the current multimodal data to be used and the event information of the pending service event into the current preset large model, and obtain the response information of the service indicated by the pending service event output by the current preset large model as the pending reply response information; wherein the current multimodal data to be used includes multiple items of the following: user tags of target customers, current weather data, current real-life data, current hot data, currently collected audio data of target customers, currently collected video data of target customers, location information of the target store, name and type of the target store, attribute information of goods in the target store, attribute information of shelves in the target store, historical order information of the target customer, and historical service data for the target customer; generate a voice corresponding to the pending reply response information as a service voice, and send the service voice to a target terminal in the target store so that the target terminal plays the service voice through a digital human.

[0063] Based on the above processing, the user tags of the target customers, the currently collected audio data and video data of the target customers can reflect the characteristics of the target customers; the historical order information of the target customers and the historical service data for the target customers can reflect the shopping preferences of the target customers; the current weather data can reflect the current environmental information; the location information of the target store can reflect the geographical characteristics of the environment where the target store is located; the current real-life data and the current hot data can reflect the current popular trends; the attribute information of the goods and shelves in the target store can reflect the layout information of the shelves of the goods in the target store. Accordingly, by processing the multimodal data using a preset large model, it is possible to generate response information (i.e., response information to be replied) that is more in line with the needs of the target customers and / or in line with the environment where the target store is located. In this way, it is possible to better provide personalized and diversified services to customers entering unmanned stores, and to improve the shopping experience of customers in unmanned stores.

[0064] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0066] Figure 1 A first flow chart of the service provision method for an unmanned store provided in an embodiment of the present application;

[0067] Figure 2 A schematic diagram of a process for generating response information to be replied based on multimodal data in a service providing method for an unmanned store provided in an embodiment of the present application;

[0068] Figure 3 A second flow chart of the service provision method for an unmanned store provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of a process of real person taking over in a service providing method for an unmanned store provided in an embodiment of the present application;

[0070] Figure 5 A schematic diagram of a process for collecting data during service event processing in a service providing method for an unmanned store provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of a first structure of a service provision system for an unmanned store provided in an embodiment of the present application;

[0072] Figure 7 A third flow chart of the service provision method for an unmanned store provided in an embodiment of the present application;

[0073] Figure 8 A second structural diagram of the service provision system for an unmanned store provided in an embodiment of the present application;

[0074] Fig. 9 A schematic diagram of the structure of a service providing device for an unmanned store provided in an embodiment of the present application;

[0075] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0077] With the development of technology, the retail industry has begun to see the emergence of cloud-based business models. Under the cloud-based business model, there are no staff on site to provide shopping guides, checkout and other services, which often requires customers to spend time and energy looking for products and figuring out settlement operations, resulting in a poor shopping experience for customers.

[0078] In order to improve the shopping experience of customers in unmanned stores, the present application embodiment provides a service provision method for unmanned stores, see Figure 1 , Figure 1 A first flow chart of a method for providing services in an unmanned store provided in an embodiment of the present application. The method includes:

[0079] Step S101: When a pending service event triggered by a target customer in a target store is detected, the multimodal data to be utilized and the event information of the pending service event are input into the current preset large model, and the response information of the service indicated by the pending service event output by the current preset large model is obtained as the pending reply response information.

[0080] Among them, the multimodal data currently to be utilized include multiple items of the following: user tags of target customers, current weather data, current real-life data, current hotspot data, currently collected audio data of target customers, currently collected video data of target customers, location information of target stores, name and type of target stores, attribute information of goods in target stores, attribute information of shelves in target stores, historical order information of target customers, and historical service data for target customers.

[0081] Step S102: Generate a voice corresponding to the response information to be replied as a service voice, and send the service voice to the target terminal in the target store, so that the target terminal plays the service voice through the digital human.

[0082] Based on the above processing, the user tags of the target customers, the currently collected audio data and video data of the target customers can reflect the characteristics of the target customers; the historical order information of the target customers and the historical service data for the target customers can reflect the shopping preferences of the target customers; the current weather data can reflect the current environmental information; the location information of the target store can reflect the geographical characteristics of the environment where the target store is located; the current real-life data and the current hot data can reflect the current popular trends; the attribute information of the goods and shelves in the target store can reflect the layout information of the shelves of the goods in the target store. Accordingly, by processing the multimodal data using a preset large model, it is possible to generate response information (i.e., response information to be replied) that is more in line with the needs of the target customers and / or in line with the environment where the target store is located. In this way, it is possible to better provide personalized and diversified services to customers entering unmanned stores, and to improve the shopping experience of customers in unmanned stores.

[0083] In addition, in the related art, the response information for the service indicated by the service event to be processed is often determined based on the preset speech configuration, and the response information obtained is relatively fixed. In the solution provided in the embodiment of the present application, the multimodal data can be processed using a preset large model to obtain more personalized and diversified response information. That is, based on the solution provided in the embodiment of the present application, it is possible to better provide personalized and diversified services to customers entering the unmanned store, and it is also possible to improve the shopping experience of customers in the unmanned store.

[0084] For steps S101 and S102, when it is detected that a target customer in a target store triggers a service event (i.e., a service event to be processed), the multimodal data to be currently utilized and the event information of the service event to be processed can be input into the current preset macro model. The event information of the service event to be processed can indicate the event type of the service event to be processed and the service content of the service indicated by the service event to be processed. For example, the event types of the service event to be processed may include: store entry events, shopping guide events, and store exit events. The service content of the service indicated by the store entry event may be the broadcast of a welcome message; the service content of the service indicated by the shopping guide event may include: searching for the attribute information of a specified commodity, the specified commodity being the commodity that the target customer currently needs to find, and the attribute information that needs to be found may include: the historical price, location, and efficacy of the specified commodity; the service content of the service indicated by the shopping guide event may also include: recommending commodities, such as recommending currently hot-selling commodities. The service content of the service indicated by the store exit event may be the broadcast of a closing message.

[0085] The current preset large model can be inferred based on the input multimodal data and event information, and output response information for the service indicated by the pending service event. For example, when it is detected that the target customer enters the target store, the pending service event triggered by the target customer can be an entry event, and the event information of the pending service event can represent the entry event. The event information representing the entry event and the current multimodal data to be used are input into the current preset large model, and the response information for the service indicated by the entry event output by the current preset large model can be obtained. The response information for the service indicated by the entry event can represent a welcome message.

[0086] When it is detected that the target customer needs a shopping guide service, the service event to be processed may be a shopping guide event, and the event information of the service event to be processed may represent the shopping guide event. The event information representing the shopping guide event and the multimodal data to be used at present are input into the current preset macro model, and the response information for the service indicated by the shopping guide event output by the current preset macro model may be obtained. The response information for the service indicated by the shopping guide event may represent the service response for the shopping guide service required by the target customer. For example, the event information representing the shopping guide event may be that the target customer needs to find the location of commodity A, and the service response for the shopping guide service required by the target customer may be that commodity A is located at the D layer of shelf C in area B of the target store.

[0087] When it is detected that the target customer has completed shopping, the service event to be processed may be a store leaving event, and the event information of the service event to be processed may indicate the store leaving event. The event information indicating the store leaving event and the multimodal data to be used are input into the current preset macro model, and the response information for the service indicated by the store leaving event output by the current preset macro model may be obtained. The response information for the service indicated by the store leaving event may indicate a conclusion.

[0088] The data included in the multimodal data to be used can be set as needed without specific limitation. The user tags of the target customers, the audio data and video data of the target customers currently collected can reflect the characteristics of the target customers. For example, with the consent of the customer, the user tags of the target customers are obtained. The user tags of the target customers can indicate the age, gender, appearance, hobbies, occupation of the target customers, and whether the target customers are new customers or old customers. In combination with the characteristics of the target customers, personalized response information can be generated. The historical order information of the target customers and the historical service data for the target customers can reflect the shopping preferences of the target customers. When the target customers need to recommend products, products that meet the shopping preferences of the target customers can be recommended.

[0089] The current weather data can reflect the current environmental information, and combined with the current environmental information, it can generate response information that is more in line with the environment. The current real-life data and the current hot data can reflect the current trends. For example, the current current affairs data can be sports events with high popularity; the current hot data can be hot events with high popularity, such as events on the hot search rankings. When the target customer needs to recommend products, it is also possible to recommend products that meet the target customer's shopping preferences and current trends. For example, combined with the current current affairs data, the recommended products can be snacks and drinks needed when watching sports events.

[0090] The location information of the target store can reflect the regional characteristics of the environment where the target store is located. For example, the regional characteristics can indicate the dialect used in the environment where the target store is located and consumption habits. In combination with the regional characteristics of the environment where the target store is located, response information that conforms to the language and consumption habits of the environment where the target store is located can be generated. The attribute information of the goods and shelves in the target store can reflect the layout information of the shelves of the goods in the target store. For example, the attribute information of the goods can indicate the name, type, quantity, price, placement and other information of the goods, and the attribute information of the shelves can indicate the location of the shelves, placement space and other information. In combination with the attribute information of the goods and shelves in the target store, shopping guide services can be provided to target customers more accurately.

[0091] Accordingly, by processing the multimodal data using a preset large model, it is possible to generate response information (i.e., response information to be replied) that is more in line with the needs of the target customers and / or the environment in which the target store is located. Furthermore, a voice corresponding to the response information to be replied (i.e., a service voice) can be generated, and the service voice can be sent to the target terminal in the target store. For example, the target terminal may be a smart screen, such as an ARM (an advanced reduced instruction set architecture) architecture, implemented based on the RK3588 module, and the digital human service implemented based on this architecture is portable and universal. The target terminal can play the service voice through the digital human.

[0092] When receiving the service voice, the target terminal can drive and render the digital human according to the preset digital human model and service voice. For example, the digital human is driven and rendered according to the preset digital human model and service voice, and the rendering result obtained can be a video frame, and the lip shape movement of the digital human represented by the rendering result matches the service voice. The preset digital human model can be set as needed, that is, the image of the digital human in the target terminal can be set as needed without specific limitation. For example, the image of the digital human can be a digital driven image of a brand spokesperson, or other images of different styles. Furthermore, the target terminal can display the digital human and play the service voice, and the lip shape movement of the displayed digital human matches the service voice. In this way, it is possible to better provide personalized and diversified services to customers entering unmanned stores, which can also improve the service quality and the shopping experience of customers in unmanned stores, and improve customer satisfaction.

[0093] In one embodiment, before inputting the multimodal data to be used and the event information of the service event to be processed into the current preset macro model, and obtaining the response information of the service indicated by the service event to be processed output by the current preset macro model as the response information to be replied, the method further includes:

[0094] The reasoning end searches for response information for the service indicated by the service event to be processed from a preset knowledge base to obtain standard response information.

[0095] Inputting the current multimodal data to be utilized and the event information of the service event to be processed into the current preset large model, obtaining the response information of the service indicated by the service event to be processed output by the current preset large model as the response information to be replied, including:

[0096] The standard response information, the event information of the service event to be processed, and the multimodal data to be used are input into the current preset large model, and combined with the random factors, the response information of the service indicated by the service event to be processed output by the current preset large model is obtained as the response information to be replied.

[0097] In an embodiment of the present application, a plurality of standard response information (also referred to as preset response information) are preset for each service event and stored in a preset knowledge base. For example, the standard response information for a store entry event may include: "Hello, welcome" and "Welcome to the store to shop", etc. The preset response information may be pre-blocked and vectorized and stored in the preset knowledge base in the form of vectors. Based on RAG (Retrieval Augmented Generation) technology, the standard response information of the service event to be processed (i.e., the standard response information for the service indicated by the service event to be processed) may be retrieved from the preset knowledge base.

[0098] A random factor may be set in the preset large model, and the random factor may be used to control the randomness and diversity of text generation. The preset large model may combine the input event information of the pending service event, the retrieved standard response information, and the input multimodal data currently to be used to obtain the response information (i.e., the response information to be replied) for the service indicated by the pending service event. When the input data remains unchanged, based on the effect of the random factor, the preset large model may output different response information, which can also improve the diversity of the response information output by the preset large model.

[0099] See also Figure 2 , Figure 2 A schematic diagram of a process for generating response information to be replied based on multimodal data in a service provision method for an unmanned store provided in an embodiment of the present application. Taking the store entry event as an example, the reasoning end (also referred to as the reasoning system) collects customer characteristics, that is, the reasoning end obtains the multimodal data currently to be used. The reasoning end inputs the standard response information of the store entry event, the multimodal data currently to be used, and the event information of the store entry event into the current preset large model, and infers to obtain a welcome message. The multimodal data currently to be used may include multiple items of the following: the customer's occupational information, the current time information, the current weather information, the customer's historical information (i.e., the customer's historical order information), the results obtained by behavioral analysis (i.e., the event information of the service event to be processed), and the shelf information (i.e., the attribute information of the shelf in the target store). The multimodal data involving the customer are all obtained with the customer's consent.

[0100] The default information indicates the standard response information, i.e., the default welcome message. For example, the default welcome message may be "Hello, welcome". If there is no historical order in the customer's historical information (i.e., the customer's historical order information), it can be determined that the customer is a new customer; otherwise, it can be determined that the customer is an old customer (also known as a familiar customer). The welcome message for new customers may be the default welcome message, and the welcome message for familiar customers may be "You are here again".

[0101] In one scenario, based on the customer's appearance, it is determined that the customer is a company employee, and the customer's occupation information is obtained. And the time information indicates nighttime. Combining the occupation information and time information, the reasoning end can infer the welcome message: "Thank you for your hard work overtime." In another scenario, the time information indicates that it is noon in summer, and the weather information indicates that the temperature is high. Combining the time information and weather information, the reasoning end can infer the welcome message: "It's hot, you've worked hard." In another scenario, based on the customer's historical information, it is determined that the customer is a regular customer, and the customer's historical orders often buy X products. The customer's behavior analysis indicates that he needs to recommend products. Based on the shelf information, the location of X product is determined to be changed. The reasoning end can infer the welcome message: "You are here again. The location of the X product you often buy has been changed to the Y shelf Z layer." Figure 2In this example, we only take the generation of a welcome message for the store entry event as an example. In the service process of digital humans, you can refer to the generation of service responses for shopping guide events or the generation of closing messages for store exit events. Figure 2 The process shown.

[0102] Based on the above processing, based on the standard response information stored in the preset knowledge base, the accuracy of the generated response information to be replied can be improved. Combined with random factors, the randomness and diversity of the generated response information to be replied can be improved. In this way, it is possible to generate more accurate and richer response information to be replied, and based on the retrieval enhancement generation technology (also called search enhancement algorithm) and random factors, the response information obtained by each reasoning can be closer to the real person's tone, which can further ensure that personalized and diversified services can be better provided to customers entering unmanned stores, and the shopping experience of customers in unmanned stores can be improved.

[0103] In one embodiment, see Figure 3 , Figure 3 A second flow chart of the service provision method of an unmanned store provided in an embodiment of the present application. The service provision method of an unmanned store is applied to a server, which includes: a data processing terminal, an inference terminal, and a speech synthesis terminal; the inference terminal is deployed with a current preset large model; and a collection device is provided in the target store for collecting video data and / or audio data in the target store.

[0104] Before step S101, the method further includes:

[0105] Step S103: The data processing end obtains the currently collected data from the collection device, performs behavior recognition on the obtained data, determines the event information of the service event to be processed triggered by the target customer, and sends the determined event information of the service event to be processed to the inference end.

[0106] Step S101 includes:

[0107] Step S1011: When the event information of the service event to be processed is received from the data processing end, the inference end inputs the multimodal data to be used and the event information of the service event to be processed into the current preset large model, obtains the response information of the service indicated by the service event to be processed output by the current preset large model as the response information to be replied, and sends the response information to be replied to the speech synthesis end.

[0108] Step S102 includes:

[0109] Step S1021: the speech synthesis terminal generates a speech corresponding to the received response information to be replied as a service speech, and sends the service speech to the target terminal in the target store, so that the target terminal plays the service speech through the digital human.

[0110] In the embodiment of the present application, the service provision method of the unmanned store can be performed by a server including a data processing terminal, an inference terminal and a speech synthesis terminal. The server can be a cloud platform that can communicate with the acquisition device set in the target store and the target terminal. For example, the acquisition device set in the target store can be a camera and a microphone, and the number and position of the camera and the microphone can be set as needed without specific limitation.

[0111] The data processing end can perform behavior recognition on the data obtained from the collection device to determine the event information of the pending service event triggered by the target customer. For example, the data processing end can determine the behavior of the target customer based on a preset behavior recognition algorithm, and then determine the pending event triggered by the target customer. For example, when the data processing end recognizes that the target customer enters the target store, it can be determined that the pending event triggered by the target customer is an entry event. In one implementation, the data processing end can be deployed with a behavior recognition model, and the data processing end can use the behavior recognition model to perform behavior recognition on the data obtained from the collection device. The data processing end can send the determined event information of the pending service event to the inference end.

[0112] The current preset large model can be deployed on the inference end. Accordingly, when receiving the pending service event sent by the data processing end, the inference end can input the current multimodal data to be used and the event information of the pending service event into the current preset large model, obtain the response information of the service indicated by the pending service event output by the current preset large model as the pending response information, and send the pending response information to the speech synthesis end. Furthermore, the speech synthesis end can generate the speech corresponding to the received pending response information as the service speech.

[0113] Based on the above processing, the server can use the preset large model to process the multimodal data to generate response information that better meets the needs of target customers and / or the environment of the target store. In this way, it is possible to better provide personalized and diversified services to customers entering the unmanned store, and improve the shopping experience of customers in the unmanned store.

[0114] In one embodiment, step S1021 includes: the speech synthesis end generates, based on a sound cloning algorithm, a speech corresponding to the response information to be replied represented by a specified timbre as a service speech.

[0115] In the embodiment of the present application, the designated timbre can be set according to the needs without specific limitation. For example, the designated timbre can match the image of the digital person. For example, if the image of the digital person is the digital driven image of the brand spokesperson, the designated timbre can be the timbre of the brand spokesperson. The speech synthesis end can generate the voice corresponding to the reply information to be replied represented by the designated timbre based on the sound cloning algorithm as the service voice. In this way, the timbre of the service voice can be set according to the needs, further improving the shopping experience of customers in unmanned stores.

[0116] In one embodiment, when the target customer is identified to enter the target store from the video data acquired from the acquisition device, the event information of the service event to be processed is determined to represent the store entry event, and the service voice represents the welcome message. When the audio data acquired from the acquisition device represents the shopping guide service request of the target customer, the event information of the service event to be processed is determined to represent the shopping guide event, and the service voice represents the service response to the shopping guide service request.

[0117] In an embodiment of the present application, the data processing end can perform behavior recognition on the video data obtained from the acquisition device. When it is determined that the target customer enters the target store, it can be determined that the event information of the service event to be processed represents an entry event. Accordingly, the response information for the service indicated by the entry event and the voice corresponding to the response information can represent a welcome message. In one implementation, the acquisition device in the target store may also include a sensor. For example, the sensor may include a door magnet, a door lock, an infrared sensor, and a radar sensor. The state of the door or the state of the entrance of the target store can be monitored in real time by the sensor. Accordingly, the data processing end can also determine that a customer has entered the target store based on the acquired sensor data, and can determine that the customer is a target customer in combination with the video data acquired from the acquisition device.

[0118] The data processing end can also identify the audio data obtained from the acquisition device. When it is determined that the audio data represents the shopping guide service request of the target customer, it can be determined that the event information of the service event to be processed represents the shopping guide event. For example, the target customer can say to the digital human "Help me find the location of product A", then the audio data can also represent that the target customer said "Help me find the location of product A", and the data processing end can determine that the event information of the service event to be processed represents that the target customer needs to find the location of product A. Correspondingly, the response information for the service indicated by the shopping guide event and the voice corresponding to the response information can represent the service response to the shopping guide service request. For example, the shopping guide service request indicates that the target customer needs to find the location of product A, and the service response to the shopping guide service request can represent the location information of product A.

[0119] Based on the above processing, the data processing end can determine the service event indicating the service required by the target customer (i.e., the service event to be processed) according to the data obtained from the collection device. And for different types of service events, different response information and corresponding voice can be obtained. In this way, it is possible to better provide personalized and diversified services to customers entering the unmanned store, and also improve the shopping experience of customers in the unmanned store.

[0120] In one embodiment, before step S1011, the method further includes:

[0121] Step 1: When a target customer is identified as entering a target store from the video data obtained from the acquisition device, the data processing end performs feature recognition on the video data obtained from the acquisition device, obtains a user tag of the target customer, and sends the obtained user tag to the inference end.

[0122] and / or,

[0123] Step 2: The inference end obtains the customer identification carried in the door opening request triggered by the target customer using the user terminal to the target store, and obtains the user tag corresponding to the obtained customer identification from the preset database.

[0124] The user tags in the preset database include: tags obtained based on registration information when a customer represented by each customer ID registers using a user terminal, and / or tags obtained based on historical purchase records of each customer ID.

[0125] In an embodiment of the present application, when a target customer is identified as entering a target store, the data processing end may perform feature recognition on the video data acquired from the acquisition device to obtain a user tag of the target customer. For example, the data processing end may use a feature recognition model to recognize the video data acquired by the acquisition device to obtain a user tag of the target customer. For example, the age, gender, appearance, and other information of the target customer may be recognized. The data processing end may send the obtained user tag of the target customer to the inference end.

[0126] The target customer may use the user terminal to trigger a door opening request for the target store. For example, the user terminal may be a mobile device carried by the target customer, such as a smart phone, a tablet computer, etc. The target store may be provided with an identification code generated based on its store logo, such as a barcode, a QR code, etc. The target customer may use the user terminal to scan the identification code of the target store to trigger a door opening request for the target store.

[0127] The target customer may register in advance on the user terminal. Specifically, the target customer may register an account in an application running on the user terminal. Then, after the target customer's permission is granted, the user terminal may trigger a door opening request for the target store through the application, and the door opening request may carry a customer identifier. For example, the customer identifier may be the mobile phone number used by the target customer when registering an account or the account number obtained by registration.

[0128] When registering, the target customer can also set the information of his or her own account (i.e., registration information), such as age, gender, hobbies, occupation, etc. The above information is recorded with the customer's consent. Accordingly, the preset database can record the customer identification of the target customer and the label obtained based on the registration information of the target customer. After the target customer purchases goods in the target store, he or she can also record the goods purchased to obtain a historical purchase record. Based on the target customer's historical purchase record, a label of the target customer can be generated and stored in the preset database. If the target customer has registered an account in advance through the user terminal, the inference end can obtain the user label corresponding to the customer identification from the preset database based on the customer identification carried in the door opening request, and obtain the user label of the target customer.

[0129] Based on the above processing, when the target customer enters the target store, the target customer can be identified by the collected video data containing the target customer to obtain the user tag of the target customer. The user tag corresponding to the customer identification of the target customer can also be obtained in the preset database. In this way, the inference end can obtain a relatively rich user tag of the target customer, and then, the inference end can also combine the relatively rich user tags to generate response information that better meets the needs of the target customer. In this way, it can ensure that personalized services can be provided to customers entering the unmanned store, further improving the shopping experience of customers in the unmanned store.

[0130] In one embodiment, sending the determined event information of the service event to be processed to the inference end includes:

[0131] Step 1: When the event information of the determined service event to be processed indicates a shopping guide event and the conditions for real-person takeover are not met, a first message containing the event information of the determined service event to be processed is sent to the on-duty terminal used by the staff of the target store, so that the on-duty terminal displays the event information of the determined service event to be processed, and sends the event information of the determined service event to the inference end.

[0132] Among them, the conditions for real-person takeover include at least one of the following: the number of conversation rounds between the digital human and the target customer is greater than a preset number, the data processing end detects that the target customer has negative emotions based on the data obtained from the collection device, and there is no service information of the service indicated by the determined service event to be processed in the preset knowledge base.

[0133] The method also includes:

[0134] Step 2: After generating the pending reply response information for the service indicated by the pending service event, the reasoning end sends the pending reply response information to the on-duty terminal, so that the on-duty terminal displays the pending reply response information.

[0135] Before step S1021, the method further includes:

[0136] Step 3: When the event information of the determined service event to be processed indicates a shopping guide event and the conditions for real-person takeover are met, the data processing end sends a second message containing the event information of the determined service event to be processed to the on-duty terminal, so that the on-duty terminal displays the event information of the determined service event to be processed, and when the staff enters the pending reply information for the service indicated by the pending service event in the on-duty terminal based on the event information and reply information of the displayed service event, the pending reply response information is sent to the speech synthesis end.

[0137] In the embodiment of the present application, a real person takeover condition may be preset. The data processing end may also determine whether the real person takeover condition is met. Specifically, the data processing end may determine whether the number of conversation rounds between the digital person and the target customer is greater than a preset number. If the number of conversation rounds between the digital person and the target customer is greater than the preset number, that is, the target customer triggers the service event multiple times, for example, the preset number may be 3 or 4. In this case, the reasoning end determines that the response information of the pending service event triggered by the target customer may make the target customer dissatisfied.

[0138] The data processing end can also detect whether the target customer has negative emotions based on the data obtained from the acquisition device. For example, the data processing end can detect the audio data obtained from the acquisition device. When it is detected that the target customer's voice becomes louder or the tone is hurried, it can be determined that the target customer has negative emotions; or the data processing end can also detect the video data obtained from the acquisition device. When it is detected that the target customer's face shows an angry expression, it can be determined that the target customer has negative emotions. In this case, the reasoning end determines that the response information of the pending service event triggered by the target customer may make the target customer dissatisfied.

[0139] The preset knowledge base may pre-store service information of services indicated by various service events. For example, the service information indicated by a shopping guide event may include location information of various commodities, the service information indicated by an entry event may include a preset welcome message, and the service information indicated by an exit event may include a preset closing message. The data processing end may also determine whether the service information of the service indicated by the determined service event to be processed exists in the preset knowledge base. For example, the service event to be processed indicates a shopping guide event, and the indicated service is that the target customer needs to find the location of commodity B, but the location information of commodity B does not exist in the preset knowledge base. It can be determined that the service information of the service indicated by the determined service event to be processed does not exist in the preset knowledge base. In this case, the reasoning end cannot determine the response information of the service event to be processed triggered by the target customer.

[0140] The terminal used by the staff of the target store can be called an on-duty terminal. When the event information of the determined pending service event indicates a shopping guide event and the conditions for real-person takeover are not met, the data processing end can send a first message to the on-duty terminal. The first message includes the event information of the determined pending service event, and the first message can instruct the on-duty terminal to display the determined pending service event and send the event information of the determined pending service event to the inference end. The inference end can determine the response information for the service indicated by the received pending service event, and can send the determined response information to the on-duty terminal. The on-duty terminal can display the received response information.

[0141] When the determined pending service event represents the event information of a shopping guide event and the conditions for real person takeover are met, it may not be possible to accurately determine the response information of the pending service event triggered by the target customer through the reasoning end. In this case, the data processing end can send a second message to the on-duty terminal. The second message contains the event information of the determined pending service event, and the second message can instruct the on-duty terminal to display the event information of the determined pending service event, and notify the staff to process the pending service event represented by the latest event information currently displayed by the on-duty terminal. The on-duty terminal can display the event information and corresponding response information of historical service events, as well as the latest received pending service event.

[0142] The staff can input the response information to be replied for the service indicated by the pending service event in the on-duty terminal according to the event information and response information of the service event displayed on the on-duty terminal. For example, a text input control can be provided in the on-duty terminal, and the staff can trigger the text input control to input the response information to be replied in the form of text in the on-duty terminal; or a voice input control can be provided in the on-duty terminal, and the staff can trigger the voice input control to input the response information to be replied in the form of voice in the on-duty terminal. The on-duty terminal can send the response information to be replied to the speech synthesis terminal.

[0143] Based on the above processing, a real person takeover condition can be set. If the real person takeover condition is not met, the inference end can use the preset large model to obtain the response information to be replied. If the real person takeover condition is met, a message can be sent to the on-duty terminal to notify the staff to handle the pending service event, and the response information to be replied can be entered in the on-duty terminal. In this way, it can be ensured that the pending service events triggered by the target customers can be handled accurately and timely, further improving the shopping experience of customers in unmanned stores.

[0144] In one embodiment, see Figure 4 , Figure 4 A flowchart of real-person takeover in the service provision method of an unmanned store provided in an embodiment of the present application.

[0145] Step S401: The digital human starts the service, that is, the digital human in the target terminal is started.

[0146] Step S402: Audio and video AI (Artificial Intelligence) recognition. That is, the data processing end obtains the currently collected data from the collection device and recognizes the obtained data.

[0147] Step S403: Determine whether the digital human dialogue exceeds 3 rounds, whether the content exceeds the scope of the knowledge base, and whether negative emotions are recognized. If yes, execute step S405; if no, execute step S404. That is, determine whether the conditions for real person takeover are met.

[0148] Step S404: Inference synthesis, that is, step 1 in the above embodiment.

[0149] Step S405: notify the duty center to take over and display the historical dialogue information. That is, the data processing end sends a second message containing the event information of the determined service event to be processed to the duty terminal, so that the duty terminal displays the event information of the determined service event to be processed. And the duty terminal can display the event information and response information of the historical service event.

[0150] Step S406: The duty center takes over and directly sends the voice message. That is, the staff inputs the response information to be replied to the service indicated by the pending service event in the duty terminal according to the event information and response information of the displayed service event.

[0151] Step S407: The smart screen digital human is rendered by voice. That is, when receiving the service voice, the target terminal can drive and render the digital human according to the preset digital human model and service voice, so that the lip movements of the displayed digital human match the service voice.

[0152] Based on the above processing, the digital human service can be taken over by the on-duty staff to improve the service quality. And the digital human rendering can be performed on the smart screen side, and the digital human can display the voice corresponding to the response information obtained by reasoning or input by the staff. The user is unaware of the process of generating response information, which can make the digital human's autonomous service and being taken over by the staff smoother. When the staff takes over, there is no need to deploy the inference end on the cloud for inference, which can save cloud computing resources. Through end-side rendering, there is no need for cloud rendering, which can further save cloud computing resources and reduce the cost of digital human rendering.

[0153] In one embodiment, the method further comprises:

[0154] For each service event triggered by a target customer, the data processing end sends the associated information of the service event to the optimization processing end, so that the optimization processing end updates the recognition model currently deployed in the data processing end for recognition based on the received associated information; wherein the associated information of any service event includes: event information of the service event, and data collected by the collection device when the target customer triggers the service event;

[0155] and / or,

[0156] For each service event triggered by the target customer, the inference end sends the input data and output data of the current preset large model in the process of processing the service event to the optimization processing end, so that the optimization processing end updates the current preset large model based on the data sent by the inference end.

[0157] In an embodiment of the present application, a recognition model for identification may be deployed in the data processing end. The data processing end may use the recognition model to perform behavior recognition on the data acquired from the acquisition device to determine the event information of the service event triggered by the target customer. The recognition model may be referred to as a behavior recognition model. For each service event triggered by the target customer, the data processing end may send the associated information of the service event to the optimization processing end (also referred to as a model training system). The associated information of any service event includes: the event information of the service event, and the audio data and / or video data collected by the acquisition device when the target customer triggers the service event. The optimization processing end may optimize the training of the current behavior recognition model based on the received associated information to update the behavior recognition model, and may synchronize the updated behavior recognition model to the data processing end.

[0158] Since the event information of the service event triggered by the target customer determined by the behavior recognition model is not necessarily accurate, before the optimization processing end optimizes and trains the current behavior recognition model, the staff responsible for the optimization processing end can review the associated information sent by the received data processing end and set a label to indicate whether the event information determined by the behavior recognition model is accurate. For example, the set label can indicate whether the service event represented by the data obtained from the collection device is a service event determined by using the behavior recognition model to perform behavior recognition on the data obtained from the collection device. If so, the label can indicate that the data obtained by the behavior recognition model is accurate.

[0159] When training the model, the optimization processing end can optimize the behavior recognition model using only event information with accurate label representation. For example, the optimization processing end can input the data collected by the acquisition device when the target customer triggers the service event represented by the event information into the behavior recognition model to obtain the predicted event information output by the behavior recognition model. Based on the difference between the predicted event information and the event information, the model parameters of the behavior recognition model are adjusted to update the behavior recognition model. Alternatively, the event information with accurate label representation can be used as a positive sample, and the event information with inaccurate label representation can be used as a negative sample to optimize the behavior recognition model.

[0160] A feature recognition model for identifying user tags may also be deployed in the data processing end. When a target customer is identified as entering a target store from the video data acquired from the acquisition device, the data processing end may use the feature recognition model to perform feature recognition on the video data acquired from the acquisition device to obtain the user tag of the target customer. The data processing end may also send the video data acquired from the acquisition device and the user tag of the target customer to the optimization processing end. The optimization processing end may optimize and train the current feature recognition model based on the data received from the data processing end to update the feature recognition model, and may synchronize the updated feature recognition model to the data processing end.

[0161] Before the optimization processing end optimizes and trains the current feature recognition model, the staff responsible for the optimization processing end can review the data sent by the data processing end and set a label to indicate whether the user label of the target customer obtained by the feature recognition model is accurate. For example, the set label can indicate whether the user label of the customer represented by the data obtained from the acquisition device is the user label obtained by using the feature recognition model to perform feature recognition on the data obtained from the acquisition device. If so, the label can indicate that the data obtained by the feature recognition model is accurate.

[0162] When performing model training, the optimization processing end may optimize the feature recognition model using only user tags with accurate label representation. For example, the optimization processing end may input the video data obtained from the acquisition device into the feature recognition model to obtain the predicted user tags. Based on the difference between the predicted user tags and the user tags of the target customers, the model parameters of the feature recognition model are adjusted to update the feature recognition model. Alternatively, the user tags with accurate label representation may be used as positive samples, and the user tags with inaccurate label representation may be used as negative samples to optimize the feature recognition model.

[0163] For each service event triggered by the target customer, the inference end sends the input data and output data of the current preset large model in the process of processing the service event to the optimization processing end. The optimization processing end can optimize and train the current preset large model based on the data sent by the inference end to update the preset large model, and can synchronize the updated preset large model to the inference end. For example, the input data of the preset large model may include: standard response information, multimodal data, and event information of the service event to be processed, and the output data of the preset large model may include: response information to be replied.

[0164] Before the optimization processing end optimizes and trains the current preset large model, the staff responsible for the optimization processing end can review the data sent by the inference end and set a label to indicate whether the response information to be replied obtained through the preset large model is accurate. For example, the set label can indicate whether the output data (i.e., the response information to be replied) obtained by using the input data of the current preset large model is the response information for the service indicated by the service event to be processed. If so, the label can indicate that the data obtained through the preset large model is accurate.

[0165] When performing model training, the optimization processing end may optimize and train the preset large model using only the response information with accurate label representation. The optimization processing end may input the standard response information, multimodal data, and the event information of the determined service event to be processed into the current preset large model to obtain the predicted response information. Based on the difference between the predicted response information and the response information to be replied, the model parameters of the current preset large model are adjusted to update the preset large model. Alternatively, the response information with accurate label representation may be used as a positive sample, and the response information with inaccurate label representation may be used as a negative sample to optimize and train the preset large model.

[0166] For each service event triggered by the target customer, the target terminal can send the voice played by the digital human in the process of processing the service event to the optimization processing end. The on-duty terminal can send the interface content displayed by itself in the process of processing the service event to the optimization processing end. Accordingly, the staff responsible for the optimization processing end can determine whether the voice played by the digital human is accurate, and whether the event information and response information displayed by the on-duty terminal in the process of processing the service event are accurate. The target terminal can also send the interface content displayed by itself in the process of processing the service event to the optimization processing end. Accordingly, the staff responsible for the optimization processing end can browse the interface content sent by the target terminal to verify the effect of the target terminal driving and rendering the digital human according to the preset digital human model and service voice.

[0167] Based on the above processing, the behavior recognition model, feature recognition model and preset large model can be updated, and the updated behavior recognition model and feature recognition model can be synchronized to the data processing end, and the updated preset large model can be synchronized to the inference end, that is, the behavior recognition model, feature recognition model and preset large model can be adaptively iterated and optimized. And for each unmanned store, at least one of the behavior recognition model, feature recognition model and preset large model corresponding to the unmanned store can be trained through the data of the customer service process in the unmanned store. In this way, the accuracy of behavior recognition using the behavior recognition model can be improved, the accuracy of feature recognition using the feature recognition model can be improved, and the accuracy of response information generated using the preset large model can be improved, further ensuring that personalized and diversified services can be better provided to customers entering the unmanned store, and the shopping experience of customers in the unmanned store can be improved.

[0168] In one embodiment, see Figure 5 , Figure 5 A flow chart of collecting data during service event processing in a service providing method for an unmanned store provided in an embodiment of the present application.

[0169] Step S501: Digital human starts service. That is, the digital human in the target terminal starts, triggering the execution of steps S502-S505.

[0170] Step S502: triggering audio and video acquisition and recording. That is, for each service event triggered by a target customer, the data processing end sends the data collected by the acquisition device when the target customer triggers the service event to the optimization processing end.

[0171] Step S503: triggering inference traffic recording. That is, for each service event triggered by the target customer, the inference end sends the input data and output data of the current preset large model in the process of processing the service event to the optimization processing end.

[0172] Step S504: triggering the screen recording of the digital human on the smart screen. That is, for each service event triggered by the target customer, the target terminal can send the voice played by the digital human during the process of processing the service event to the optimization processing end.

[0173] Step S505: triggering the screen recording and intercom recording of the on-duty client. That is, for each service event triggered by the target customer, the on-duty terminal can send the interface content displayed by itself during the process of processing the service event to the optimization processing end.

[0174] Based on the above processing, for each service event triggered by the target customer, the optimization processing end can obtain the data collected by the collection device when the target customer triggers the service event, the input data and output data of the current preset large model in the process of processing the service event, the voice played by the digital human, and the interface content displayed by the on-duty terminal. And the behavior recognition model, feature recognition model and preset large model can be iterated and upgraded according to the acquired data. In this way, the accuracy of behavior recognition using the behavior recognition model can be improved, the accuracy of feature recognition using the feature recognition model can be improved, and the accuracy of response information generated using the preset large model can be improved, further ensuring that personalized and diversified services can be better provided to customers entering the unmanned store, and the shopping experience of customers in the unmanned store can be improved.

[0175] In one embodiment, see Figure 6 , Figure 6 A first structural diagram of the service provision system for an unmanned store provided in an embodiment of the present application. The system may include: an inference and synthesis system 601, a remote service system 602, a smart screen side 603, a merchant management system 604, a media data acquisition side 605, and a media data processing system 606. The inference and synthesis system 601 includes a cloud synthesis system 6011 (i.e., the speech synthesis end in the above embodiment), and an inference system 6012 (i.e., the inference end in the above embodiment). The remote service system 602 includes a remote service center, i.e., the duty center in the above embodiment. The smart screen side 603 includes a smart screen terminal, i.e., the target terminal in the above embodiment. The merchant management system 604 includes a merchant backend, i.e., the merchant management end in the subsequent embodiments. The media data acquisition side 605 includes cameras and sensors, i.e., the acquisition devices in the above embodiment. Figure 6 Only three cameras and one sensor are used as an example, and the number of cameras and sensors can be set as needed. The media data processing system 606 is the data processing end in the above embodiment, including multimedia services for forwarding audio data and / or video data, and an AI recognition algorithm. The AI ​​recognition algorithm is the behavior recognition model and feature recognition model in the above embodiment.

[0176] The media data collection side 605 can collect audio and video for customers, and upload real-time audio and video to multimedia services. The multimedia service can forward audio and video in real time, that is, forward audio and video to the remote service center and AI recognition algorithm. The AI ​​recognition algorithm can obtain behavioral events (i.e., pending service events) based on the acquired data recognition, and input the determined behavioral events into the reasoning system 6012 through the remote service center. That is, the data processing end obtains the currently collected data from the acquisition device, recognizes the acquired data, and can send the determined pending service events to the on-duty terminal, and the on-duty terminal sends the determined pending service events to the reasoning end. The reasoning system 6012 can infer the response to the behavioral event, send the response to the cloud synthesis system 6011, and synchronize the response to the remote service center. That is, execute steps 1 and 2 in the above embodiment. A real person can be taken over by the remote service center, that is, execute step 3 in the above embodiment.

[0177] The cloud synthesis system 6011 can generate a service voice based on the received response information, based on the sound cloning and synthesis algorithm, and send the audio (i.e., the service voice) to the smart screen terminal. That is, the voice synthesis end generates a voice corresponding to the response information to be replied represented by a specified timbre based on the sound cloning algorithm as a service voice. The smart screen terminal can drive and render the digital human according to the preset digital human model and service voice so that the lip movements of the displayed digital human match the service voice. Customers can have voice intercoms with the smart screen terminal. The merchant backend can be used to manage product SKU (StockKeeping Unit, the minimum inventory unit, i.e., product identification), inventory, pricing, shelves, orders, payments and other information. That is, the merchant management end can record the attribute information of the goods in the target store, the attribute information of the shelves in the target store, and the historical order information of the customers of the target store, and synchronize the recorded information to the inference end, i.e., upload the recorded information to the inference end. And the smart screen terminal can obtain and display the product and order information from the merchant backend to realize the display of products and orders.

[0178] In one embodiment, see Figure 7 , Figure 7 A third flow chart of the service provision method of an unmanned store provided in an embodiment of the present application. The method may include:

[0179] Step S701: A camera or sensor collects a customer entering a store event, that is, a target customer in a target store is detected to trigger a store entry event.

[0180] Step S702: Start the digital human, that is, start the digital human in the target terminal and start providing services.

[0181] Step S703: the media device starts collecting media data, that is, the data processing end can obtain the currently collected data from the collection device.

[0182] Step S704: The AI ​​analysis service obtains customer characteristics based on media data and other data analysis. That is, the data processing end performs feature recognition and behavior recognition on the video data obtained from the acquisition device. The user tag of the target customer can be obtained, and the inference end can also obtain the user tag corresponding to the customer identifier from the preset database.

[0183] Step S705: The inference system infers the welcome message by combining customer characteristics, time, location, space, current events and other multi-modal data, that is, step S1011 in the above embodiment.

[0184] Step S706: The synthesis system synthesizes the speech according to the cloned timbre and sends it down, that is, step S1021 in the above embodiment.

[0185] Step S707: The smart screen plays the welcome message and drives the digital human through voice. That is, when receiving the service voice, the target terminal can drive and render the digital human according to the preset digital human model and service voice, so that the lip movements of the displayed digital human match the service voice.

[0186] The embodiment of the present application provides a solution for providing personalized services in unmanned stores, combines multimodal information to implement service and shopping guide functions, and improves the duty efficiency and service satisfaction in unmanned stores.

[0187] Based on the same inventive concept, the present application also provides a service provision system for unmanned stores, see Figure 8 , Figure 8 A second structural diagram of the service providing system for an unmanned store provided in an embodiment of the present application, the system comprising: a service end 801 and a target terminal 802 in a target store;

[0188] The server 801 is used to execute the service providing method of the unmanned store described in any of the above embodiments;

[0189] The target terminal 802 is used to receive the service voice sent by the service end and play the service voice through the digital human.

[0190] Based on the service provision system of the unmanned store provided by the embodiment of the present application, the user tag of the target customer, the audio data and video data of the target customer currently collected can reflect the characteristics of the target customer; the historical order information of the target customer and the historical service data for the target customer can reflect the shopping preferences of the target customer; the current weather data can reflect the current environmental information; the location information of the target store can reflect the regional characteristics of the environment where the target store is located; the current facts data and the current hot data can reflect the current popular trends; the attribute information of the goods and shelves in the target store can reflect the layout information of the shelves of the goods in the target store. Correspondingly, by processing the multimodal data using a preset large model, it is possible to generate response information (i.e., response information to be replied) that is more in line with the needs of the target customer and / or in line with the environment where the target store is located. In this way, it is possible to better provide personalized and diversified services to customers entering the unmanned store, and it is also possible to improve the shopping experience of customers in the unmanned store.

[0191] In one embodiment, the system further comprises: an on-duty terminal used by a staff member of the target store;

[0192] The on-duty terminal is used to, when receiving a first message containing event information of a determined service event to be processed sent by the data processing end, display the event information of the determined service event to be processed, and send the event information of the determined service event to be processed to the reasoning end; when receiving response information to be replied sent by the reasoning end, display the received response information to be replied;

[0193] When a second message containing event information of a determined service event to be processed is received from the data processing end, the event information of the determined service event to be processed is displayed, and when the staff inputs the pending reply information for the service indicated by the pending service event in the on-duty terminal based on the event information and reply information of the displayed service event, the pending reply information is sent to the speech synthesis end.

[0194] In one embodiment, the system further includes: an optimization processing end;

[0195] The target terminal is further used to send the voice played by the digital human in the process of processing each service event triggered by the target customer to the optimization processing end;

[0196] The on-duty terminal is further used to send the interface content displayed by itself during the process of processing each service event triggered by the target customer to the optimization processing end;

[0197] The optimization processing end is used to update the recognition model currently deployed in the data processing end for recognition based on the associated information of each service event received from the data processing end, and / or to update the current preset large model based on the input data and output data of the current preset large model in the process of processing each service event received from the inference end.

[0198] In one embodiment, the system further comprises: a collection device provided in the target store;

[0199] The acquisition device is used to acquire video data and / or audio data in the target store.

[0200] In one embodiment, the system further includes: a merchant management terminal;

[0201] The merchant management end is used to record the attribute information of the goods in the target store, the attribute information of the shelves in the target store, and the historical order information of the customers of the target store, and synchronize the recorded information to the inference end.

[0202] In the embodiment of the present application, a merchant management terminal may also be provided, and the merchant management terminal may record the attribute information of the goods and shelves in the target store and the historical order information of the target customer. And the staff may manage the recorded information through the merchant management terminal, for example, the attribute information of the goods and shelves may be updated through the merchant management terminal, such as checking the attribute information of the goods, replenishing the goods in time, updating the attribute information of the goods, and synchronizing the updated information to the inference terminal; or, the order information of the target customer may be recorded, and the order information of the target customer may be sent to the target terminal 802, and the target terminal 802 may display the received order information, and accordingly, the target customer may browse the order information displayed in the target terminal 802 and perform payment operations. In this way, the shopping experience of customers in unmanned stores may be further guaranteed.

[0203] In one embodiment, the target terminal 802 is specifically configured to drive and render the digital human according to a preset digital human model and the service voice when receiving the service voice, so that the lip movements of the displayed digital human match the service voice.

[0204] In the technical solution of this application, the operations involved in obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0205] Based on the same inventive concept, the present application also provides a service providing device for an unmanned store, see Fig. 9 , Fig. 9A schematic diagram of a service providing device for an unmanned store provided in an embodiment of the present application. The device includes:

[0206] The response information determination module 901 is used to input the current multimodal data to be utilized and the event information of the service event to be processed into the current preset macro model when a service event to be processed is detected to be triggered by a target customer in a target store, and obtain the response information of the service indicated by the service event to be processed output by the current preset macro model as the response information to be replied;

[0207] The multimodal data to be used currently includes multiple items of the following: user tags of the target customer, current weather data, current real-time data, current hot spot data, currently collected audio data of the target customer, currently collected video data of the target customer, location information of the target store, name and type of the target store, attribute information of goods in the target store, attribute information of shelves in the target store, historical order information of the target customer, and historical service data for the target customer;

[0208] The service voice generation module 902 is used to generate a voice corresponding to the response information to be replied as a service voice, and send the service voice to the target terminal in the target store so that the target terminal plays the service voice through the digital human.

[0209] Based on the service providing device of the unmanned store provided by the embodiment of the present application, the user tag of the target customer, the audio data and video data of the target customer currently collected can reflect the characteristics of the target customer; the historical order information of the target customer and the historical service data for the target customer can reflect the shopping preferences of the target customer; the current weather data can reflect the current environmental information; the location information of the target store can reflect the regional characteristics of the environment where the target store is located; the current facts data and the current hot data can reflect the current popular trends; the attribute information of the goods and shelves in the target store can reflect the layout information of the shelves of the goods in the target store. Correspondingly, by processing the multimodal data using a preset large model, it is possible to generate response information (i.e., response information to be replied) that is more in line with the needs of the target customer and / or in line with the environment where the target store is located. In this way, it is possible to better provide personalized and diversified services to customers entering the unmanned store, and it is also possible to improve the shopping experience of customers in the unmanned store.

[0210] The present application also provides an electronic device, such as Fig.10 As shown, including:

[0211] Memory 1001, used for storing computer programs;

[0212] The processor 1002 is used to implement the following steps when executing the program stored in the memory 1001:

[0213] When a service event to be processed is detected, which is triggered by a target customer in a target store, the multimodal data to be used and the event information of the service event to be processed are input into the current preset macro model, and the response information of the service indicated by the service event to be processed output by the current preset macro model is obtained as the response information to be replied;

[0214] The multimodal data to be used currently includes multiple items of the following: user tags of the target customer, current weather data, current real-time data, current hot spot data, currently collected audio data of the target customer, currently collected video data of the target customer, location information of the target store, name and type of the target store, attribute information of goods in the target store, attribute information of shelves in the target store, historical order information of the target customer, and historical service data for the target customer;

[0215] A voice corresponding to the response information to be replied is generated as a service voice, and the service voice is sent to a target terminal in the target store, so that the target terminal plays the service voice through a digital human.

[0216] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1002, the communication interface, and the memory 1001 communicate with each other via the communication bus.

[0217] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0218] The communication interface is used for communication between the above electronic device and other devices.

[0219] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0220] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0221] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned unmanned store service providing methods are implemented.

[0222] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any unmanned store service providing method in the above-mentioned embodiments.

[0223] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0224] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0225] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system, device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0226] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for providing services in an unmanned store, characterized in that: The method comprises: When a service event to be processed is detected, which is triggered by a target customer in a target store, the multimodal data to be used and the event information of the service event to be processed are input into the current preset macro model, and the response information of the service indicated by the service event to be processed output by the current preset macro model is obtained as the response information to be replied; The multimodal data to be used currently includes multiple items of the following: user tags of the target customer, current weather data, current real-time data, current hot spot data, currently collected audio data of the target customer, currently collected video data of the target customer, location information of the target store, name and type of the target store, attribute information of goods in the target store, attribute information of shelves in the target store, historical order information of the target customer, and historical service data for the target customer; A voice corresponding to the response information to be replied is generated as a service voice, and the service voice is sent to a target terminal in the target store, so that the target terminal plays the service voice through a digital human.

2. The method according to claim 1, characterized in that The method is applied to a server, which includes: a data processing end, an inference end and a speech synthesis end; the inference end is deployed with a current preset large model; the target store is provided with a collection device for collecting video data and / or audio data in the target store; When a service event to be processed is detected to be triggered by a target customer in a target store, the method further includes: inputting the multimodal data to be used and the event information of the service event to be processed into the current preset macro model, and obtaining the response information of the service indicated by the service event to be processed output by the current preset macro model as the response information to be replied. The data processing end acquires the currently acquired data from the acquisition device, performs behavior recognition on the acquired data, determines the event information of the service event to be processed triggered by the target customer, and sends the determined event information of the service event to be processed to the reasoning end; When a service event to be processed is detected and triggered by a target customer in a target store, the multimodal data to be used and the event information of the service event to be processed are input into the current preset macro model, and response information for the service indicated by the service event to be processed output by the current preset macro model is obtained as the response information to be replied, including: When receiving the event information of the pending service event sent by the data processing end, the reasoning end inputs the multimodal data to be used and the event information of the pending service event into the current preset large model, obtains the response information of the service indicated by the pending service event output by the current preset large model as the pending reply response information, and sends the pending reply response information to the speech synthesis end; The generating of the voice corresponding to the response information to be replied as the service voice includes: The speech synthesis end generates speech corresponding to the received response information to be replied as a service speech.

3. The method according to claim 2, characterized in that In the case where the target customer is identified to enter the target store from the video data acquired by the acquisition device, the event information of the service event to be processed determined to represent an entry event, and the service voice represents a welcome message; In the case where the audio data acquired from the acquisition device represents the shopping guide service request of the target customer, the event information of the service event to be processed determined represents a shopping guide event, and the service voice represents a service response to the shopping guide service request.

4. The method according to claim 3, characterized in that When receiving the event information of the service event to be processed sent by the data processing end, the reasoning end inputs the multimodal data to be currently utilized and the event information of the service event to be processed into the current preset large model, and obtains the response information of the service indicated by the service event to be processed output by the current preset large model as the response information to be replied, the method further includes: When the target customer is identified as entering the target store from the video data acquired from the acquisition device, the data processing end performs feature recognition on the video data acquired from the acquisition device to obtain a user tag of the target customer, and sends the obtained user tag to the inference end; and / or, The inference end obtains the customer identifier carried in the door opening request for the target store triggered by the target customer using the user terminal, and obtains the user tag corresponding to the obtained customer identifier from a preset database; wherein the user tags in the preset database include: tags obtained based on registration information when the customer represented by each customer identifier registers using the user terminal, and / or tags obtained based on historical purchase records of each customer identifier.

5. The method according to claim 3, characterized in that: Sending the determined event information of the service event to be processed to the inference end includes: In the case where the event information of the determined service event to be processed indicates a shopping guide event and the condition for real person taking over is not met, a first message containing the event information of the determined service event to be processed is sent to the on-duty terminal used by the staff of the target store, so that the on-duty terminal displays the event information of the determined service event to be processed, and sends the event information of the determined service event to be processed to the reasoning end; wherein the real person taking over condition includes at least one of the following: the number of dialogue rounds between the digital human and the target customer is greater than a preset number, the data processing end detects that the target customer has negative emotions based on the data obtained from the acquisition device, and the service information of the service indicated by the determined service event to be processed does not exist in the preset knowledge base; The method further comprises: After generating the pending reply response information for the service indicated by the pending service event, the reasoning end sends the pending reply response information to the on-duty terminal so that the on-duty terminal displays the pending reply response information; Before the speech synthesis terminal generates the speech corresponding to the received response information to be replied as the service speech, the method further includes: When the event information of the determined service event to be processed indicates a shopping guide event and the conditions for real-person takeover are met, the data processing end sends a second message containing the event information of the determined service event to be processed to the on-duty terminal, so that the on-duty terminal displays the event information of the determined service event to be processed, and when the staff inputs the pending reply information for the service indicated by the pending service event in the on-duty terminal based on the event information and reply information of the displayed service event, the pending reply information is sent to the speech synthesis end.

6. The method according to claim 2, characterized in that The method further comprises: For each service event triggered by the target customer, the data processing end sends the associated information of the service event to the optimization processing end, so that the optimization processing end updates the recognition model currently deployed in the data processing end for recognition based on the received associated information; wherein the associated information of any service event includes: event information of the service event, and data collected by the collection device when the target customer triggers the service event; and / or, For each service event triggered by the target customer, the inference end sends the input data and output data of the current preset large model in the process of processing the service event to the optimization processing end, so that the optimization processing end updates the current preset large model based on the data received from the inference end.

7. The method according to claim 2, characterized in that The speech synthesis end generates a speech corresponding to the received response information to be replied as a service speech, including: The speech synthesis end generates, based on a voice cloning algorithm, a speech corresponding to the response information to be replied represented by a specified timbre as a service speech.

8. The method according to any one of claims 1 to 7, characterized in that: Before inputting the current multimodal data to be utilized and the event information of the service event to be processed into the current preset macro model, and obtaining the response information of the service indicated by the service event to be processed output by the current preset macro model as the response information to be replied, the method further includes: The reasoning end searches for response information for the service indicated by the to-be-processed service event from a preset knowledge base to obtain standard response information; The step of inputting the multimodal data to be currently utilized and the event information of the service event to be processed into the current preset macro model, and obtaining the response information of the service indicated by the service event to be processed output by the current preset macro model as the response information to be replied includes: The standard response information, the event information of the service event to be processed, and the multimodal data to be used are input into the current preset large model, and combined with the random factor, the response information of the service indicated by the service event to be processed output by the current preset large model is obtained as the response information to be replied.

9. A service provision system for unmanned stores, characterized in that: The system comprises: a service end and a target terminal in a target store; The server is used to execute the service providing method of the unmanned store according to any one of claims 1 to 8 above; The target terminal is used to receive the service voice sent by the service end and play the service voice through the digital human.

10. The system according to claim 9, characterized in that The system also includes: an on-duty terminal used by staff of the target store; The on-duty terminal is used to, when receiving a first message containing event information of a determined service event to be processed sent by the data processing end, display the event information of the determined service event to be processed, and send the event information of the determined service event to be processed to the reasoning end; when receiving response information to be replied sent by the reasoning end, display the received response information to be replied; When a second message containing event information of a determined service event to be processed is received from the data processing end, the event information of the determined service event to be processed is displayed, and when the staff inputs the pending reply information for the service indicated by the pending service event in the on-duty terminal based on the event information and reply information of the displayed service event, the pending reply information is sent to the speech synthesis end.

11. The system according to claim 10, characterized in that The system further comprises: an optimization processing end; The target terminal is further used to send the voice played by the digital human in the process of processing each service event triggered by the target customer to the optimization processing end; The on-duty terminal is further used to send the interface content displayed by itself during the process of processing each service event triggered by the target customer to the optimization processing end; The optimization processing end is used to update the recognition model currently deployed in the data processing end for recognition based on the associated information of each service event received from the data processing end, and / or to update the current preset large model based on the input data and output data of the current preset large model in the process of processing each service event received from the inference end.

12. The system according to any one of claims 9 to 11, characterized in that: The system further comprises: a collection device provided in the target store; The acquisition device is used to acquire video data and / or audio data in the target store.

13. The system according to any one of claims 9 to 11, characterized in that: The system also includes: a merchant management terminal; The merchant management end is used to record the attribute information of the goods in the target store, the attribute information of the shelves in the target store, and the historical order information of the customers of the target store, and synchronize the recorded information to the inference end.

14. The system according to any one of claims 9 to 11, characterized in that: The target terminal is specifically configured to drive and render the digital human according to a preset digital human model and the service voice when receiving the service voice, so that the lip movements of the displayed digital human match the service voice.

15. A service providing device for an unmanned store, characterized in that: The device comprises: A response information determination module is used to input the current multimodal data to be utilized and the event information of the service event to be processed into the current preset macro model when a service event to be processed is detected to be triggered by a target customer in a target store, and obtain the response information of the service indicated by the service event to be processed output by the current preset macro model as the response information to be replied; The multimodal data to be used currently includes multiple items of the following: user tags of the target customer, current weather data, current real-time data, current hot spot data, currently collected audio data of the target customer, currently collected video data of the target customer, location information of the target store, name and type of the target store, attribute information of goods in the target store, attribute information of shelves in the target store, historical order information of the target customer, and historical service data for the target customer; The service voice generation module is used to generate the voice corresponding to the response information to be replied as the service voice, and send the service voice to the target terminal in the target store so that the target terminal plays the service voice through the digital human.

16. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

18. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 8.

Images