Method and system for generating dynamic image in real time based on dialogue content and electronic equipment

Abstract

The invention provides a method and a system for generating a dynamic image in real time based on dialogue content, and electronic equipment. Determining interaction parameters such as an interaction theme, an emotional tendency quantized value, interaction time, an interaction position and an interaction role corresponding to the voice signal and an interaction task preset according to the interaction position, and then determining a visual element combination matched with each parameter in the interaction parameters according to the interaction parameters; and generating a moving dynamic image based on each visual object unit designed in advance according to each interaction parameter in the interaction scene in the visual element combination. Mapping the dynamic image to an LED array used for displaying, and controlling the state change of the corresponding LED light-emitting units in the LED array according to the pixel position of the dynamic image. By means of the method and device, man-machine interaction conversation around participants in the interaction process can be achieved, matched dynamic images are generated in an accompanying mode, and more immersive interaction experience is provided for the participants.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, system and electronic device for generating dynamic images in real time based on conversation content. Background Art

[0002] With the development of artificial intelligence technology, the application scenarios of human-computer dialogue have been widely used in all aspects of daily life. Existing human-computer dialogue scenarios mainly focus on generating and displaying text content based on the content of human-computer dialogue. For some scenarios that rely on visual images for content understanding, such as cultural and tourism scenarios, collection content introductions, theme parks, etc., the existing human-computer dialogue methods can only generate text content and lack the ability to generate richer visual feedback elements, and cannot provide users with a more immersive use experience and interactive experience. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a method, system and electronic device for generating dynamic images in real time based on conversation content, so as to realize real-time generation of dynamic images according to the conversation content of human-computer interaction, thereby providing users with a more immersive usage experience and interactive experience.

[0004] In a first aspect, an embodiment of the present application provides a method for generating a dynamic image in real time based on a conversation content, wherein the method comprises:

[0005] Acquire an input voice signal, and determine interaction parameters corresponding to the voice signal, wherein the interaction parameters include: interaction topic, emotional tendency quantification value, interaction time, interaction position, interaction role, and interaction tasks pre-set according to the interaction position;

[0006] Determine a visual element combination that matches the interaction parameters, and generate a dynamic image based on the visual element combination, wherein the visual element combination includes a plurality of visual elements, and the visual elements are visual object units designed in advance according to the interaction parameters in the interaction scene;

[0007] The dynamic image is mapped onto an LED array, and the state change of the corresponding LED light-emitting unit in the LED array is controlled according to the pixel position of the dynamic image.

[0008] In some possible embodiments, determining a combination of visual elements matching the interaction parameter includes:

[0009] The interaction parameters are input into a visual element matching model, and the visual element combination is constructed based on each visual element output by the visual element matching model, wherein the visual element matching model is a data model trained in advance based on the matching relationship between the interaction theme content label, the emotion tendency label and each visual element as sample data.

[0010] In some possible embodiments, generating a dynamic image based on the combination of visual elements includes:

[0011] Determine two visual elements G displayed adjacent to each other in the visual element combination 01 , G 02 ;

[0012] Based on the visual element G 01 Each pixel d 01 With visual elements G 02 Each corresponding pixel d 02 , by using the preset image deformation algorithm, calculate the deformed pixel point d within time t t ;

[0013] Based on the pixel point d 01 , the pixel point d 02 And the deformed pixel point d t , combined with the playback order to generate the dynamic image.

[0014] In some possible embodiments, the method further includes:

[0015] The pixel point d is interpolated by a preset color interpolation algorithm. 01 , the pixel point d 02 And the deformed pixel point d t Perform linear interpolation processing to generate dynamic color images.

[0016] In some possible embodiments, the method further includes:

[0017] Initial music matching the user's identity tag information is obtained, and when the interactive task starts, the initial music is played, and the state change of the LED light-emitting unit is controlled based on the pixel coordinate value and color value of each pixel point corresponding to the dynamic image.

[0018] In some possible embodiments, mapping the dynamic image onto the LED array includes:

[0019] Using a nearest neighbor interpolation algorithm, based on a mapping relationship between a first coordinate range of the dynamic image and a second coordinate range of the display image of the LED array, determine a target pixel point position corresponding to each pixel point of the dynamic image in the display image of the LED array;

[0020] According to the pixel coordinate value and color value corresponding to the position of the target pixel point, the brightness, color and flickering of the LED light-emitting unit corresponding to the target pixel point are controlled.

[0021] In some possible embodiments, the method further includes:

[0022] If no voice signal is input beyond a preset time, the LED array is controlled to enter an energy-saving mode, wherein the energy-saving mode includes: randomly determining a target visual element for display according to a first set display time interval, and controlling the LED array to display the target visual element within a second set display time interval.

[0023] In a second aspect, the present application provides a system for generating dynamic images in real time based on conversation content, the system comprising:

[0024] An input module, used to obtain an input voice signal and determine interaction parameters corresponding to the voice signal, wherein the interaction parameters include: interaction topic, emotional tendency quantification value, interaction time, interaction position, interaction role, and interaction tasks pre-set according to the interaction position;

[0025] An image generation module, used to determine a visual element combination matching the interaction parameters, and generate a dynamic image based on the visual element combination, wherein the visual element combination includes a plurality of visual elements, and the visual elements are visual object units pre-designed according to the interaction parameters in the interaction scene;

[0026] The image display module is used to map the dynamic image onto the LED array and control the state change of the corresponding LED light-emitting unit in the LED array according to the pixel position of the dynamic image.

[0027] In a third aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes: a processor; and a memory for storing programs; wherein the program includes instructions, which, when executed by the processor, enable the processor to execute the method for real-time generation of dynamic images based on conversation content as described in the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the method of generating dynamic images in real time based on conversation content as described in the first aspect.

[0029] Beneficial effects of this application:

[0030] The present application provides a method, system and electronic device for generating dynamic images in real time based on the content of a conversation, wherein the method obtains an input voice signal, determines the interaction parameters such as the interaction theme, emotional tendency quantization value, interaction time, interaction position, interaction role and interaction task pre-set according to the interaction position corresponding to the voice signal, and then determines a visual element combination matching each parameter in the interaction parameter according to the interaction parameter, and generates a dynamic image based on each visual object unit in the visual element combination that is pre-designed according to each interaction parameter in the interaction scene. Then, the dynamic image is mapped to an LED array for display, and the state change of the corresponding LED light-emitting unit in the LED array is controlled according to the pixel position of the dynamic image.

[0031] The embodiment of the present application can create an overall experience of LED array dynamic image playback in the background of human-computer dialogue. Combined with the personalized image design of interactive scenes and interactive tasks, it can realize the human-computer interactive dialogue around the participants during the interactive process, and generate matching dynamic images in a companion manner. For some scenes that require interactive experience such as amusement parks, theme parks, exhibition halls, etc., the embodiment of the present application can provide users with a more immersive use experience and interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 A schematic diagram of a process of a method for generating a dynamic image in real time based on a conversation content provided by an embodiment of the present application is shown;

[0034] Figure 2 A schematic diagram of a logical architecture of a system for generating dynamic images in real time based on conversation content provided by an embodiment of the present application is shown;

[0035] Figure 3 Another logical architecture diagram of a system for generating dynamic images in real time based on conversation content provided by an embodiment of the present application is shown;

[0036] Figure 4 A structural block diagram of an exemplary electronic device that can be used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0038] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0039] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0040] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0041] In order to realize the real-time generation of dynamic images adapted to the current interactive scene based on the content of human-computer dialogue, so as to provide users with a more immersive use experience and interactive experience. The present application provides a method, system and electronic device for generating dynamic images in real time based on the content of dialogue, wherein the method for generating dynamic images in real time based on the content of dialogue provided by the present application can be applied to any electronic device capable of generating real-time dynamic images based on the content of dialogue. Similarly, the system for generating dynamic images in real time based on the content of dialogue provided by the present application can be deployed in the electronic device capable of generating real-time dynamic images based on the content of dialogue. The type of the electronic device may vary according to the application scenario, including but not limited to: personal mobile terminal devices, computers or servers, etc.

[0042] As an example, the electronic device can be a trigger device in an amusement theme park for providing interactive services to tourists. The trigger device can be a portable interactive device, such as a bracelet or a prop, or it can be a fixed interactive device set up in the amusement park, such as a fixed touch button or a touch screen.

[0043] In the first aspect, the present application provides a method for generating dynamic images in real time based on conversation content, such as Figure 1 As shown, the method comprises the following steps:

[0044] S11, obtaining an input voice signal, and determining interaction parameters corresponding to the voice signal, wherein the interaction parameters include: interaction theme, emotional tendency quantification value, interaction time, interaction position, interaction role, and interaction tasks pre-set according to the interaction position;

[0045] S12, determining a visual element combination that matches the interaction parameter, and generating a dynamic image based on the visual element combination, wherein the visual element combination includes a plurality of visual elements, and the visual elements are visual object units designed in advance according to the interaction parameters in the interaction scene;

[0046] S13, mapping the dynamic image onto an LED array, and controlling the state change of corresponding LED light-emitting units in the LED array according to pixel positions of the dynamic image.

[0047] The embodiment of the present application obtains the input voice signal, determines the interaction parameters such as the interaction theme, emotional tendency quantization value, interaction time, interaction position, interaction role and interaction task pre-set according to the interaction position corresponding to the voice signal, and then determines the visual element combination matching each parameter in the interaction parameter according to the interaction parameter, and generates a dynamic image based on each visual object unit in the visual element combination that is pre-designed according to each interaction parameter in the interaction scene. Then, the dynamic image is mapped to the LED array for display, and the state change of the corresponding LED light-emitting unit in the LED array is controlled according to the pixel position of the dynamic image.

[0048] By selecting the embodiment of the present application, an overall experience of LED array dynamic image playback in the background of human-computer dialogue can be created. Combined with the personalized image design of interactive scenes and interactive tasks, it is possible to achieve human-computer interactive dialogue around the participants during the interactive process, and to generate matching dynamic images in an accompanying manner, providing participants with a more immersive interactive experience.

[0049] To facilitate understanding of the beneficial effects of the embodiments of the present application, this article takes the method provided by the embodiments of the present application applied to an amusement park as an example. It is assumed that there are several attractions in an amusement park, and different attractions form a complete tour route. If tourists arrive at an attraction on the tour route, they can interact with the interactive device in the attraction to complete an interactive mini-game specially designed for the attraction. There is continuity between the plots of the interactive mini-games between the various attractions. The interactive mini-games are interactive events. Different attractions have different types of interactive events, and the interactive events of various attractions constitute a complete story.

[0050] As the executor of the embodiment of the present application, the interactive device can determine how many interactive events the current tourist has completed, the interactive events that need to be completed, the theme corresponding to the current tourist attraction, the emotional tendencies of the tourist at this time, and the tourist's response based on the type of interactive event by interacting with tourists who have arrived at the tourist attraction where the interactive device is located, thereby generating dynamic images that match the current tourist attraction, the current theme, the emotional tendencies of the current tourist, and the type of interactive event, thereby increasing the tourist's visual interactive experience and thus enhancing the tourist's travel experience.

[0051] The following will describe the above steps S11 to S13 in detail with reference to specific examples:

[0052] Wherein, when executing step S11, the input voice signal is obtained, which may be a voice signal collected by a microphone module inside the acquisition device, such as the execution subject of the present application, and the input voice signal is a voice signal collected by the microphone of the interactive device. As another embodiment, the input voice signal can be obtained by acquiring a voice signal collected by an external microphone device and sent to the execution subject, wherein the external sound pickup device and the execution subject of the method of the embodiment of the present application are not the same device, and the two devices are connected via wireless communication. Exemplarily, the two devices are connected via Bluetooth.

[0053] In the embodiment of the present application, the device capable of collecting voice signals is referred to as a sound pickup device. Based on this, in some embodiments, since the application scenario of the method provided in the embodiment of the present application may be a large public place, the place may include other sounds besides the conversation users, that is, there may be noise interference. In order to ensure the accuracy of the generated sound effect, during the execution of the above step S11, the method further includes:

[0054] S11-1. Acquire a voice signal collected by a target sound pickup device, and add identity tag information to the voice signal.

[0055] S11-2. Based on a preset noise reduction algorithm, the voice signal of the target sound pickup device is subjected to noise reduction processing to obtain a noise-reduced voice signal to be analyzed.

[0056] Among them, when executing step S11-1, the identity of the interacting user and his / her interactive role information can be determined by interacting with other identity recognition devices in the application scenario. Exemplarily, the identity information of the interacting user and the role information R associated with the interacting user can be determined through near field communication, such as NFC, RFID, etc., bracelet sensing, or face recognition.

[0057] In an embodiment of the present application, the interacting user is closest to the sound collection device and has the highest intensity among the stably collected sounds. The conversation voice between the user and the interactive device is acquired in real time, and a close-range sound pickup device is used in hardware, in conjunction with a pre-set noise reduction algorithm, to ensure that only one user's voice is recognized as a conversation with the interactive device at the same time. The preset noise reduction algorithm performs spectral analysis on the collected audio signal to identify and filter out environmental noise and other irrelevant sound frequency components, thereby improving the accuracy of speech recognition. Among them, the specific type of the preset noise reduction algorithm can be set according to the actual application scenario, and this application does not strictly limit it.

[0058] For example, assuming that the input speech signal is S voice (t), t represents time. Through the preset AI noise reduction algorithm, S voice (t) is converted into the noise-reduced speech signal S' to be analyzed voice (t).

[0059] Furthermore, when executing step S11, in determining the interaction parameters corresponding to the voice signal, the interaction location can be set according to the location where the interaction behavior occurs, which can be the location obtained by GPS positioning within the execution subject, or the location of the device interacting with it. Similarly, the interaction time corresponding to the voice signal can be determined based on the system time of the execution subject's interaction device, wherein the system time of the interaction device has a regular calibration function to ensure the accuracy of the interaction time.

[0060] When executing step S11, the interaction theme corresponding to the voice signal can be determined according to the location of the interaction device. For example, taking the aforementioned amusement park as an example, different scenes correspond to different themes. If an interaction device is provided in each scene, the interaction theme corresponding to the interaction device that receives the voice signal can be determined according to the mapping relationship between the interaction device and the scene.

[0061] As another implementation, the interaction topic corresponding to the voice signal can be determined based on the voice content of the participant's interactive voice. Specifically, the interaction topic of the voice signal can be determined in the following manner:

[0062] S11-3, converting the input voice signal into target text content according to a preset voice-to-text function;

[0063] S11-4, performing semantic analysis based on the target text content, extracting associated words contained in the target text content and related to the preset interactive theme;

[0064] S11-5. According to the associated vocabulary, determine the interaction topic with the highest degree of association with the associated vocabulary as the interaction topic corresponding to the voice signal.

[0065] In step S11-3, the preset speech-to-text function can be any algorithm function with speech-to-text function. For example, a trained speech-to-text neural network model can be used to convert the input speech signal into the target text content, or to convert the input speech signal to be analyzed into the target text content. Assume that the preset speech-to-text function is T trans () then the speech signal to be analyzed S' voice (t) is converted into the corresponding target text content W, where W = T trans (S' voice (t)).

[0066] Further, step S11-4 is performed as an implementation method, and semantic analysis is performed on the target text content W through a preset natural language processing tool or model to extract the nouns and emotion-related auxiliary words contained in the target text content W, so as to subsequently determine the corresponding interaction theme and the emotional tendency quantification value based on the extracted nouns and emotion-related auxiliary words. As another implementation method, possible interaction themes can be pre-set based on the applicable scenarios of the method provided in the embodiment of the present application. Such pre-set interaction themes are preset interaction themes. Exemplarily, taking the aforementioned amusement park as an example, assuming that the applied amusement park is a science and technology amusement park, which is subdivided into aerospace technology, biotechnology, computer technology, quantum technology, etc., and then matching the extracted nouns with the set vocabulary is performed to extract the associated related vocabulary contained in the target text content, and further determine which science and technology theme is currently.

[0067] As another implementation, a language text recognition and classification model may be pre-trained based on a specific application scenario, such as a text classification model based on word vectors and deep learning, to determine the interactive topic with the highest degree of relevance to the conversation content. Then, step S11-5 is executed to determine the interactive topic with the highest degree of relevance to the associated vocabulary based on different associated vocabulary as the interactive topic corresponding to the voice signal. Exemplarily, taking the aforementioned amusement park as an example of an application scenario, assuming that the amusement park is divided into different plot areas, and different plot areas correspond to a theme type, the vocabulary of which plot is specifically contained in the target text can be determined based on the vocabulary associated with the amusement park plot contained in the current target text content, and the plot with the highest degree of relevance is determined as the theme type corresponding to the current voice signal.

[0068] Based on the target text content converted from the participants' voice signals, we construct the subject word library corresponding to different topics by using the vocabulary that may be involved in different topics, and then use the semantic analysis model to analyze the current interactive dialogue topic C. Specifically, assuming that the text content is C and the subject word library is B, for each word c in C i , calculate each word c i and each subject word b in B j The semantic similarity s(c i ,b j ), which can be calculated based on the word vector model i , b j Then, the frequency of occurrence of words related to each topic is counted f(c i ) and the sum of the semantic association degree, that is, the sum of the occurrence frequency and the semantic association degree S is calculated according to the following formula j :

[0069]

[0070] Then, select S j The largest topic is taken as the interaction topic corresponding to the speech signal.

[0071] As another implementation, a language text recognition and classification model can be pre-trained based on a specific application scenario, such as a text classification model based on word vectors and deep learning, to determine the interaction topic with the highest degree of relevance to the conversation content. Then, based on different associated words, the interaction topic with the highest degree of relevance to the associated words is determined as the interaction topic corresponding to the voice signal. The specific algorithm implementation in the above steps S11-4 and S11-5 can be understood as follows: Assuming that a topic classification function is obtained based on actual application scenario training: F topic (), then input the target text content into the F topic () function, the corresponding interaction topic can be obtained, that is, C = F topic(W). Specific interactive topics can be subdivided into aerospace technology, biotechnology, computer technology, quantum technology, etc.

[0072] In some possible embodiments, the emotional tendency quantification value corresponding to the speech signal can also be obtained by analyzing the interactive speech signal. As an implementation method, semantic analysis can be performed based on the target text content, and then the modal particles contained in the target text content can be extracted to further assist in judging the current emotions and emotional tendencies of the participants. Specifically, the emotional tendency quantification value can be obtained in the following way:

[0073] S11-6, obtaining audio features of the speech signal, the audio features including: speech rate v, volume l, and pitch p;

[0074] S11-7, based on the reference speech speed, reference volume, and reference pitch, normalizing the speech features to obtain a normalized audio difference ratio;

[0075] S11-8. Calculate the target emotional tendency score according to the audio difference ratio and preset emotional tendency label information, and determine the score based on the target emotional tendency score as the emotional tendency quantization value corresponding to the speech signal.

[0076] When executing step S11-6, as an implementation method, the number of words or syllables per unit time in the speech signal can be calculated as the speech rate v of the speech signal. Specifically, by identifying the word or syllable boundaries in the speech signal, then measuring the time intervals between the word or syllable boundaries, and then calculating the number of words or syllables per unit time.

[0077] As an implementation method, audio feature extraction can be performed on the speech signal to extract the amplitude of the speech signal as the volume l of the speech signal. Specifically, the corresponding volume can be calculated by calculating the sum of the amplitudes of all sampling points in a sound frame, or by calculating the sum of the squares of the amplitudes of all sampling points and then taking the logarithm as the volume of the speech signal.

[0078] As an implementation method, the basic frequency of the speech signal can be extracted by performing spectrum analysis on the speech signal through methods such as short-time Fourier transform, and then the basic frequency is determined as the pitch p of the speech signal.

[0079] In step S11-7, the reference speech speed v0, the reference volume l0, and the reference pitch p0 are the standards for measuring the speech speed, volume, and pitch of the speech signal. By comparing the speech speed v of the speech signal with the reference speech speed v0, the speed of the participant's speech can be determined. Similarly, by comparing with the reference volume, the volume of the participant's speech can be determined, and by comparing with the reference pitch, the pitch of the participant's speech can be determined. Since the speech speed, volume, and pitch are data under different dimensions, in order to facilitate unified calculation, in the embodiment of the present application, the actual speech features of the participant are normalized based on the reference amount to obtain a normalized audio difference ratio, wherein, corresponding to the audio features of the speech, the audio difference ratio also includes three categories: speech speed audio difference ratio, volume audio difference ratio, and pitch audio difference ratio.

[0080] In the embodiment of the present application, the normalization process can be performed by obtaining the difference between the actual audio feature and the reference value: the ratio between the reference values ​​as the audio difference ratio. Specifically, taking volume as an example, the volume audio difference ratio r l =(actual volume of speech signal l - reference volume l0) ÷ reference volume l0, and so on, the speech speed audio difference ratio r v =(actual speech speed v of speech signal - reference speech speed v0)÷reference speech speed v0, pitch audio difference ratio r p =(actual pitch p of the speech signal - reference pitch p0) ÷ reference pitch p0.

[0081] Further, step S11-8 is performed to calculate the target emotional tendency score according to the normalized audio difference ratio and the mapping relationship between the audio difference ratio and the preset emotional tendency label. As an implementation method, a comprehensive audio difference weighting coefficient K can be obtained by multiplying the volume audio difference ratio, the speech speed audio difference ratio, and the pitch audio difference ratio, that is, K = (1 + volume audio difference ratio r l )×(1+ratio of speech speed and audio difference v )×(1+pitch audio difference ratio r p ).

[0082] Among them, the preset emotional tendency label is a data label E that is pre-set step by step from negative to positive according to the emotional tendency. Among them, the levels corresponding to the specific data labels can be simple three levels of negative, neutral, and positive, or they can be pessimistic, relatively pessimistic, negative, relatively negative, neutral, relatively positive, positive, relatively optimistic, and optimistic, a total of 9 levels. Different levels correspond to different emotional tendency label values, that is, corresponding to different E values. In this way, the target emotional tendency score E' can be calculated according to the formula E'=K×E, and the emotional tendency score of the participant can be calculated by executing the above steps S11-6 to S11-8 when the participant first uses the service of generating real-time dynamic images based on the conversation content provided by the embodiment of the present application.

[0083] As another implementation, the speaking speed v of the participant at different times can be compared with the participant's multiple human-computer interactions. i , Volume i and the pitch p i , learn the voice characteristics of the participant, and gradually perform normalization processing, so that the calculation of the above-mentioned comprehensive audio difference ratio K can be more in line with the conventional voice characteristics of the participant, avoiding the comprehensive audio difference ratio obtained by a single calculation from excessively affecting the result of the sentiment tendency analysis, and finally update the sentiment tendency score E' in the above manner.

[0084] Specifically, the volume, speech rate, and pitch values ​​used to determine the emotional tendency can be updated by using statistical methods such as mean and standard deviation. For example, the volume mean can be calculated by calculating the mean and standard deviation. Average speaking speed Mean pitch Volume standard deviation σ l , standard deviation of speech rate σ v , standard deviation of pitch σ p , and then use the volume mean Average speaking speed Mean pitch Volume standard deviation σ l , standard deviation of speech rate σ v , standard deviation of pitch σ p Perform normalization again to get the normalized volume value r l ′, normalized speech rate value r v ′, normalized pitch value r p ′, and then use the normalized volume value r′ l , the normalized speech rate value r′ v , the normalized pitch value r′ p Calculate the final comprehensive K' to make the result of K' more reasonable. Among them, the normalized volume value r' l , the normalized speech rate value r′v , the normalized pitch value r′ p The calculation and normalization methods of are the same. Here, we take the speaking speed as an example to illustrate, and the other two items will not be repeated:

[0085] Assume that the speech rate of multiple interactive voices of participants is {v1,v2,...,v n}, and then calculate the average speaking speed The following formula can be used to calculate the

[0086]

[0087] Where n is the total number of interactive voices participating in the calculation of the average speech rate, i is the sequence number of the interactive voice participating in the calculation of the average speech rate, and then normalized by the following formula to obtain the normalized speech rate value r v ′:

[0088]

[0089] Then K' is calculated by the following formula:

[0090] K′=(1+r l ′)(1+r v ′)(1+r p ′)

[0091] In this way, it is possible to avoid the individual regular speech characteristics of the participants from excessively affecting the calculation accuracy of the emotional tendency quantization value.

[0092] In an embodiment of the present application, the interactive role corresponding to the voice signal determined in step S11 can be obtained by obtaining the participant's reply voice signal for analysis during the human-computer interaction process by asking the participant about the participant's current interactive role. As another implementation, it can be determined by obtaining the interactive role checked by the participant in the interactive interface when entering the interactive scene. Exemplarily, taking the application scenario of an amusement park as an example, when entering the amusement park, visitors can check the role they want to play, and the role checked by the visitor is the interactive role. In an embodiment of the present application, when executing step S11 to determine the interactive task corresponding to the voice signal, the corresponding interactive task can be determined by obtaining the position of the interactive device of the executing subject.

[0093] Based on the multiple interaction parameters obtained in step S11, step S12 may be further performed to determine a combination of visual elements that matches the interaction parameters. Specifically, as a possible implementation, step S12 may be implemented by the following steps:

[0094] S12-1. Input the interaction parameters into the visual element matching model, and construct the visual element combination based on the visual elements output by the visual element matching model, wherein the visual element matching model is a data model trained in advance based on the matching relationship between the interaction theme content label, the emotion tendency label and each visual element as sample data.

[0095] Specifically, each visual element can be designed and produced according to the overall environment of the interactive content and the interactive scene. The visual element can be a geometric figure, a cartoon image of an interactive character, or an animation IP (intellectual property) figure such as a person or an object, or a figure derived from different color combinations of the above-mentioned figures. With the help of the visual elements in different interactive scenes, interactive theme content labels are constructed for each visual element in advance according to different interactive scenes and interactive themes, and the interactive theme content labels are used to mark which interactive scenes the visual element is suitable for. Similarly, emotional tendency labels are constructed for each visual element in advance according to different emotional tendencies, and the emotional state the visual element is suitable for is marked by the emotional tendency labels.

[0096] Then, with the help of the interactive theme content label and the emotional tendency label, the matching relationship between each visual element and the interactive theme and the interactive emotional tendency can be reflected. Then, based on the matching relationship between the interactive theme content label, the emotional tendency label and each visual element as training sample data, the pre-set learning model is trained. The learning model is used to learn the association between the visual elements and the applied interactive scenarios and the adapted emotional states until the learning model can accurately match the adapted visual elements according to the input interactive theme and emotional tendency. At this time, the trained learning model is the visual element matching model.

[0097] Specifically, during the training process, it is assumed that the training data is: {(C1, E1, G1), (C2, E2, G2), ..., (C m ,E m ,G m )}, where C i Corresponding to the interactive topic content label, E i Corresponding to the sentiment label, G i Corresponding and interactive topic content label C i , emotional tendency label E i The model is trained with this training data to optimize the model weights, so that the trained model can accurately output the corresponding G based on the input C and E.

[0098] The position L of the interactive device, the interactive role R, the interactive time t, the interactive task O and other information can be used as input parameters of the model and input into the model, and the model retrieves the matching visual elements. The position L can be a coordinate vector (x, y, z) or an area identifier, the interactive role R can be a role type number, the interactive time t can be a timestamp or a time period identifier, and the interactive task O can be a pre-set basic event type number and other information.

[0099] Furthermore, a dynamic image is generated based on the above-obtained visual element combination through the following steps:

[0100] S12-2, determining two visual elements G displayed adjacently in the visual element combination 01 , G 02 ;

[0101] S12-3, based on the visual element G 01 Each pixel d 01 With visual elements G 02 Each corresponding pixel d 02 , by using the preset image deformation algorithm, calculate the deformed pixel point d within time t t ;

[0102] S12-4, based on the pixel point d 01 , the pixel point d 02 And the deformed pixel point d t , combined with the playback order to generate the dynamic image.

[0103] As an implementation method, assuming that the visual element matching model is M, the output visual element of the visual element matching model M is G = M (L, R, t, O), that is, the visual element output by the matching model M is derived from a set of visual elements {G 01 ,G 02 ,...,G M}.

[0104] Furthermore, the visual element G that needs to be displayed adjacently can be determined from the visual element combination. 01 , G 02 Then determine G 01 Any pixel d 01 To G 02 The corresponding pixel d 02 , by using the preset image deformation algorithm, calculate the pixel point d of two pixels at time point t during the deformation process tThe value of t can be flexibly set according to actual needs. As a preferred implementation, the value of t can be 0≤t≤1s. The preset image deformation algorithm can flexibly select an existing image deformation algorithm according to the actual application scenario requirements. This application does not strictly limit the preset image deformation algorithm.

[0105] Specifically, assuming that G 01 The coordinates of the pixel point in is d 01 (x 01 ,y 01 ), the corresponding G 02 The coordinates of the pixel point in is d 02 (x 02 ,y 02 ), with the help of image deformation algorithm, the deformed pixel point d is calculated according to the following formula t The coordinate x t ,y t :

[0106]

[0107] Furthermore, the changing sequence of the entire dynamic image can be understood as the change from the visual element G 01 The pixel d in 01 , to deform pixel d t As a transition pixel, it changes to the visual element G 02 The pixel d in 02 In this way, a dynamic image d can be generated based on the above playback sequence. 01 —>d t —>d 02 As another implementation, the visual element frames constructed by each deformed pixel point can be used as key frames, and the key frame interpolation technology can be used to interpolate the visual element G 01 With visual elements G 02 The deformed pixels are inserted between the pixels to achieve a dynamic effect.

[0108] In the embodiment of the present application, during the execution of the above step S12, a dynamic image can also be generated through the following steps:

[0109] The pixel point d is interpolated by a preset color interpolation algorithm. 01 , the pixel point d 02 And the deformed pixel point d t Perform linear interpolation processing to calculate the coordinate values ​​and RGB color values ​​of the corresponding pixel points to generate dynamic images.

[0110] In some possible embodiments, in the process of playing the dynamic image generated in step S12 and executing step S13 to control the state change of the LED light-emitting unit, the dynamic image can also be matched with appropriate music in the following manner:

[0111] Initial music matching the user's identity tag information is obtained, and when the interactive task starts, the initial music is played, and the state change of the LED light-emitting unit is controlled based on the pixel coordinate value and color value of each pixel point corresponding to the dynamic image.

[0112] In the embodiment of the present application, the initial music that matches the user's identity tag can be obtained by matching based on a pre-trained music matching model, or when the user selects an interactive task, different interactive tasks correspond to different background music, and the background music is the initial music. Then, when the user completes the interactive task, the user's identity tag is obtained by interacting with the user, and then the interactive task and interactive role selected by the user are determined by the user's identity tag, and then the adaptive background music is determined as the initial music based on the interactive task and interactive role. In this way, when the interactive task starts to execute, while playing the initial music, the state change of the LED light-emitting unit in the LED array is controlled based on the pixel coordinate value, color value, etc. of each pixel point corresponding to the dynamic image, thereby providing the participants with a dual auditory and visual participation experience.

[0113] As another implementation, during the entire interaction process of the participants, the interactive voice signal can be sampled and analyzed according to a set sampling interval. As a preferred implementation, the set sampling time interval is △t=3s. n The text content corresponding to the speech signal sampled at the moment is C n , based on the text content, the interactive topic keyword B is analyzed n and sentiment tendency quantification value E n Then, based on the analysis results, the visual element G is generated by the visual element matching model. n =M(B n ,E n ). As a result, a dynamic switching effect can be formed between the visual elements of the previous time interval and the visual elements of the next time interval, which can help create a richer immersive and personalized participation experience.

[0114] In some possible embodiments, when executing the above step S13, the dynamic image may be mapped onto the LED array in the following manner:

[0115] S13-1, using a nearest neighbor interpolation algorithm, based on a mapping relationship between a first coordinate range of the dynamic image and a second coordinate range of the display image of the LED array, determining a target pixel point position corresponding to each pixel point of the dynamic image in the display image of the LED array;

[0116] S13-2. Control the brightness, color, and flickering of the LED light-emitting unit corresponding to the target pixel point according to the pixel coordinate value and color value corresponding to the target pixel point position.

[0117] In the process of executing the above steps S13-1 and S13-2, it can be understood that the generated dynamic image is mapped to the LED arrays with different resolutions using the nearest neighbor interpolation algorithm. Specifically, it is assumed that the coordinate range of the original dynamic image G is (0,0) to (W,H), where W is the width of the original image and H is the height of the original image. The display image of the LED array is the target image to be mapped, and the target image G d The coordinate range is (0,0)~(W d ,H d ). d is the width of the target image, H d is the height of the target image. Furthermore, for the original image and the target image, there is a corresponding mapping relationship between the pixels. Specifically, for the pixel coordinates (x, y) in the original image, the corresponding pixel coordinates (x d ,y d ) satisfies the following formula:

[0118]

[0119] Assume G 01 Middle pixel d 01 The RGB color value is (r 01 ,g 01 ,b 01 ), G 02 Middle pixel d 02 The RGB color value is (r 02 ,g 02 ,b 02 ), the deformed pixel d at time t t The RGB color value calculation satisfies the following formula:

[0120]

[0121] Furthermore, the target image G d The pixel point (x d ,y d ) The coordinate value of the actual LED light-emitting unit corresponding to xd ,y d The color value of the light-emitting unit is obtained by rounding down. d , g d , b d The rounding operation is performed.

[0122] Thus, during the execution of step S13-2, the specific states such as brightness, color, and flickering of the LED light-emitting unit can be controlled according to the calculated pixel coordinate value and color value, so as to achieve the effect of generating and displaying a matching dynamic image according to the interactive voice signal. In the embodiment of the present application, the LED light-emitting unit includes: LED lamp beads, organic light-emitting diodes OLED, and light-emitting units obtained by more different LED chip packaging methods.

[0123] In some possible embodiments, the interactive voice signal can be sampled at a shorter time and higher frequency sampling interval, and then the corresponding visual elements can be generated, so as to form a closer dynamic image matching effect with the interpersonal dialogue. In this process, assuming that the sampling function is F sample (t), at t n The text content sampled at the sampling time is C n =F sample (t n ), then based on the text content C n Matching generates visual elements G n When a certain interactive voice conversation fails to be sampled, the visual elements and dynamic display effects in the previous sampling time interval can be kept and played in a loop until the next sampling time interval during which the conversation voice signal is successfully recognized and the corresponding visual elements and dynamic display effects can be successfully generated. In this way, the negative impact on the visual experience caused by sampling failure can be reduced.

[0124] In some possible embodiments, the voice signal input may be monitored, and if there is no voice signal input beyond a preset time, the LED array is controlled to enter the energy-saving mode. Specifically, the energy-saving mode includes: a ), every t a Time from the collection of visual elements {G 01 ,G 02 ,...,G M} randomly selects visual elements for display. Then the LED array is controlled to display the second set time length (assuming that t b ), at t bThe selected visual element is displayed within the time length. The specific time lengths of the first set time and the second set time can be flexibly set according to the actual scenario. For example, assuming that no interaction occurs, a visual element in the visual element library is randomly selected every 10 seconds and displayed on the LED array, and the specific display duration is 3 seconds. In this way, it can not only save energy, but also attract participants to participate in the interaction.

[0125] In a second aspect, the present application provides a system for generating dynamic images in real time based on conversation content, wherein, Figure 2 As shown, the system 20 includes:

[0126] The input module 201 is used to obtain an input voice signal and determine the interaction parameters corresponding to the voice signal, wherein the interaction parameters include: interaction theme, emotional tendency quantization value, interaction time, interaction position, interaction role, and interaction tasks pre-set according to the interaction position;

[0127] An image generation module 202, configured to determine a visual element combination matching the interaction parameters, and generate a dynamic image based on the visual element combination, wherein the visual element combination includes a plurality of visual elements, and the visual elements are visual object units pre-designed according to the interaction parameters in the interaction scene;

[0128] The image display module 203 is used to map the dynamic image onto the LED array, and control the state change of the corresponding LED light-emitting unit in the LED array according to the pixel position of the dynamic image.

[0129] In some possible implementations, the system for generating dynamic images in real time based on conversation content provided by the present application can also be as follows: Figure 3 As shown, the method described in the first aspect is implemented specifically through the following modules:

[0130] The participant identity information recognition module 31 is used to execute the above steps S11 - 1 and S11 - 2 to recognize the identity of the participant and add identity tag information to the voice signal so as to match different visual elements according to different participants later.

[0131] The dialogue content analysis module 32 is used to execute the above steps S11 - 3 to S11 - 5 to determine the text content corresponding to the voice signal and the interactive topics included in the text content.

[0132] The emotional tendency analysis module 33 is used to execute the above steps S11-6 to S11-8 to determine the emotional tendency quantization value corresponding to the speech signal.

[0133] The visual element and dynamic image generation module 34 is used to execute the above steps S12 - 1 to S12 - 4 , determine the matching visual elements, and generate dynamic images based on the matched visual elements.

[0134] The LED array control and display output module 35 is used to execute the above step S13 and control the LED array to display based on the generated dynamic image and the interaction duration.

[0135] By selecting the embodiments of the present application, it is possible to realize accompanying and real-time generation of dynamic images that match comprehensive information such as human-computer dialogue, the location of the interactive device, the interactive role, the interactive time, and the interactive task, thereby creating an immersive and interactive participation experience.

[0136] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0137] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0138] In a third aspect, the exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present application when executed by the at least one processor.

[0139] The exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0140] The exemplary embodiments of the present application further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to execute the method according to the embodiment of the present application.

[0141] refer to Figure 4, the structural block diagram of the electronic device 400 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0142] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM 402) or a computer program loaded from a storage unit 408 into a random access memory (RAM 403). In RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, ROM 402 and RAM 403 are connected to each other via a bus 404. An input / output interface (I / O interface 405) is also connected to the bus 404.

[0143] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0144] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the aforementioned method for generating dynamic images in real time based on the content of a conversation may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the aforementioned method for generating dynamic images in real time based on the content of a conversation by any other appropriate means (e.g., by means of firmware).

[0145] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0146] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0149] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0150] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A method for generating dynamic images in real time based on conversation content, characterized in that: The method comprises: Acquire an input voice signal, and determine interaction parameters corresponding to the voice signal, wherein the interaction parameters include: interaction topic, emotional tendency quantification value, interaction time, interaction position, interaction role, and interaction tasks pre-set according to the interaction position; Determine a visual element combination that matches the interaction parameters, and generate a dynamic image based on the visual element combination, wherein the visual element combination includes a plurality of visual elements, and the visual elements are visual object units pre-designed according to the interaction parameters in the interaction scene; The dynamic image is mapped onto an LED array, and the state change of the corresponding LED light-emitting unit in the LED array is controlled according to the pixel position of the dynamic image.

2. The method according to claim 1, characterized in that The determining of a combination of visual elements matching the interaction parameters comprises: The interaction parameters are input into a visual element matching model, and the visual element combination is constructed based on each visual element output by the visual element matching model, wherein the visual element matching model is a data model trained in advance based on the matching relationship between the interaction theme content label, the emotion tendency label and each visual element as sample data.

3. The method according to claim 1, characterized in that The generating of dynamic images based on the combination of visual elements comprises: Determine two visual elements G displayed adjacent to each other in the visual element combination 01 , G 02 ; Based on the visual element G 01 Each pixel d 01 With visual elements G 02 Each corresponding pixel d 02 , by using the preset image deformation algorithm, calculate the deformed pixel point d within time t t ; Based on the pixel point d 01 , the pixel point d 02 And the deformed pixel point d t , combined with the playback order to generate the dynamic image.

4. The method according to claim 3, characterized in that The method further comprises: The pixel point d is interpolated by a preset color interpolation algorithm. 01 , the pixel point d 02 And the deformed pixel point d t Perform linear interpolation processing to generate dynamic color images.

5. The method according to claim 3, characterized in that: The method further comprises: Initial music matching the user's identity tag information is obtained, and when the interactive task starts, the initial music is played, and the state change of the LED light-emitting unit is controlled based on the pixel coordinate value and color value of each pixel point corresponding to the dynamic image.

6. The method according to claim 1, characterized in that Mapping the dynamic image onto the LED array includes: Using a nearest neighbor interpolation algorithm, based on a mapping relationship between a first coordinate range of the dynamic image and a second coordinate range of the display image of the LED array, determine a target pixel point position corresponding to each pixel point of the dynamic image in the display image of the LED array; According to the coordinate value and color value corresponding to the position of the target pixel point, the brightness, color and flickering of the LED light-emitting unit corresponding to the target pixel point are controlled.

7. The method according to claim 1, characterized in that The method further comprises: If no voice signal is input beyond a preset time, the LED array is controlled to enter an energy-saving mode, wherein the energy-saving mode includes: randomly determining a target visual element for display according to a first set display time interval, and controlling the LED array to display the target visual element within a second set display time interval.

8. A system for generating dynamic images in real time based on conversation content, characterized in that: The system comprises: An input module, used to obtain an input voice signal and determine interaction parameters corresponding to the voice signal, wherein the interaction parameters include: interaction topic, emotional tendency quantification value, interaction time, interaction position, interaction role, and interaction tasks pre-set according to the interaction position; An image generation module, used to determine a visual element combination matching the interaction parameters, and generate a dynamic image based on the visual element combination, wherein the visual element combination includes a plurality of visual elements, and the visual elements are visual object units pre-designed according to the interaction parameters in the interaction scene; The image display module is used to map the dynamic image onto the LED array and control the state change of the corresponding LED light-emitting unit in the LED array according to the pixel position of the dynamic image.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing a program; wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-7.

Images