Speech recognition method, device, equipment, medium and product

Abstract

The invention relates to a voice recognition method and device, equipment, a medium and a product, and the method comprises the steps: obtaining the lip information of a user and the voice information of a given target sound in a sound production process of the user; determining standby initial information of each word in the target sound based on the lip shape information; performing voice recognition on the voice information, obtaining text information after recognition, and determining at least one target sensitive word in the text information; and based on the standby initial consonant information and the target sensitive word, determining a voice recognition result of the target sound, and based on the voice recognition result, determining a to-be-executed task. The accuracy of the speech recognition method can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition technology, and in particular to a speech recognition method, device, equipment, medium and product. Background Art

[0002] With the rapid development of science and technology, voice recognition technology has become a common way of human-computer interaction in people's daily lives. Through voice recognition technology, electronic devices can recognize the voice information input by users and perform various tasks that users want to complete, greatly improving people's work efficiency and life enjoyment.

[0003] In the related art, electronic devices only perform speech recognition on the user's voice. If the voice recognition is performed in an environment with relatively loud noises, the speech recognition results may have large errors. In addition, speech recognition in the related art often uses a single speech recognition model without combining the user's lip shape information, which may lead to inaccurate speech recognition results. Summary of the invention

[0004] The present disclosure provides a speech recognition method, apparatus, device, medium and product.

[0005] According to a first aspect of the present disclosure, a speech recognition method is provided, the method comprising:

[0006] During the user's utterance, obtaining the user's lip shape information and the voice information of the target sound emitted;

[0007] Determine the backup initial consonant information of each word in the target sound based on the lip shape information;

[0008] Performing speech recognition on the speech information to obtain text information after the recognition, and determining at least one target sensitive word in the text information;

[0009] Based on the backup initial consonant information and the target sensitive word, a speech recognition result of the target sound is determined, and based on the speech recognition result, a task to be performed is determined.

[0010] Further, the determining of the spare initial consonant information of each word in the target sound based on the lip shape information includes:

[0011] The lip shape information is input into the improved YOLOv8 model for processing to obtain the backup initial consonant information of each word in the target sound.

[0012] Furthermore, the lip shape information is input into the improved YOLOv8 model for processing to obtain the spare initial consonant information of each word in the target sound, including:

[0013] Performing feature extraction on the lip shape information through the backbone network in the YOLOv8 model to obtain a lip shape feature map;

[0014] Input the lip shape feature map into the EMA attention mechanism module for processing to obtain a first attention matrix;

[0015] The first attention matrix is ​​subjected to feature map fusion processing through the neck module in the YOLOv8 model, and the fusion result is detected through the head module in the YOLOv8 model to obtain the backup initial consonant information of each word in the target sound.

[0016] Furthermore, the performing feature map fusion processing on the first attention matrix through the neck module in the YOLOv8 model includes:

[0017] Inputting the first attention matrix into the SPPF network in the neck module for processing to obtain a first processing result;

[0018] Processing the first processing result through a MSAM multi-scale convolutional attention module to obtain a second processing result;

[0019] The second processing result is subjected to feature fusion processing through the remaining network layers in the neck module to obtain a fusion result.

[0020] Further, performing speech recognition on the speech information to obtain text information after recognition, and determining at least one target sensitive word in the text information includes:

[0021] Inputting the voice information into a pre-built voice recognition model to obtain the text information;

[0022] At least one target sensitive word is identified in the text information through a decision tree.

[0023] Further, the identifying at least one target sensitive word in the text information through a decision tree includes:

[0024] Determine the occurrence position and frequency of each of the text information in the voice information;

[0025] Inputting the occurrence position and the occurrence frequency into the decision tree for processing to obtain the word sensitivity of the text information;

[0026] When the word sensitivity of the text information is greater than a preset sensitivity threshold, the text information is determined to be the target sensitive word.

[0027] Further, the determining the speech recognition result of the target sound based on the backup initial consonant information and the target sensitive word includes:

[0028] When the initial consonant set includes the initial consonants of each sensitive word in the target sensitive word, determining the speech recognition result based on the target sensitive word;

[0029] In the case that the initial consonant set does not include at least some of the initial consonants of the sensitive characters, a speech recognition result is determined based on the initial consonant set; wherein the initial consonant set includes the backup initial consonant information.

[0030] Further, determining the task to be performed based on the speech recognition result includes:

[0031] Matching the speech recognition result with the speech tasks in the speech task library to obtain a matching degree between the speech task and the speech recognition result;

[0032] The voice task whose matching degree meets a preset condition is determined as the task to be executed.

[0033] According to a second aspect of the present disclosure, a speech recognition device is provided, the device comprising:

[0034] An acquisition module, used to acquire the lip shape information of the user and the voice information of the target sound emitted during the user's utterance;

[0035] A first determination module, used for determining the spare initial consonant information of each word in the target sound based on the lip shape information;

[0036] A second determination module is used to perform speech recognition on the speech information, obtain text information after recognition, and determine at least one target sensitive word in the text information;

[0037] The third determination module is used to determine the speech recognition result of the target sound based on the backup initial consonant information and the target sensitive word, and determine the task to be performed based on the speech recognition result.

[0038] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0039] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0040] According to a fifth aspect of the present disclosure, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the method of the present disclosure is implemented.

[0041] The embodiments of the present disclosure provide a speech recognition method, apparatus, device, medium and product. In the embodiments of the present application, first, during the user's speech, the user's lip shape information and the speech information of the target sound emitted are obtained; then, the backup initial consonant information of each word in the target sound is determined based on the lip shape information; thereafter, speech recognition is performed on the speech information, and text information is obtained after recognition, and at least one target sensitive word is determined in the text information; finally, the speech recognition result of the target sound is determined based on the backup initial consonant information and the target sensitive word, and the task to be performed is determined based on the speech recognition result.

[0042] From the above description, it can be seen that the present disclosure obtains the user's lip shape information and voice information respectively during the user's voice utterance, and determines the backup initial consonant information and target sensitive words based on the lip shape information and voice information respectively, and finally determines the voice recognition result through the backup initial consonant information and the target sensitive word, thereby determining the task to be executed. The present disclosure realizes the method of combining the user's lip shape information and voice information to determine the voice recognition result. Compared with the traditional technology of only performing voice recognition on the user's voice, the voice recognition result determined by the present disclosure is more accurate and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0044] Figure 1 A flowchart of a speech recognition method provided for an exemplary embodiment of the present disclosure;

[0045] Figure 2 A schematic diagram of the structure of an improved YOLOv8 model provided by an exemplary embodiment of the present disclosure;

[0046] Figure 3 A flowchart of a speech recognition method provided for another exemplary embodiment of the present disclosure;

[0047] Figure 4 A flowchart of a speech recognition method provided for another exemplary embodiment of the present disclosure;

[0048] Figure 5 A schematic block diagram of functional modules of a speech recognition device provided by an exemplary embodiment of the present disclosure;

[0049] Figure 6 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0050] Figure 7 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure;

[0051] Figure 8 A structural block diagram of a computer program product provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0053] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0054] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0055] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0056] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0057] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0058] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0059] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0060] In one embodiment, Figure 1 As shown, a speech recognition method is provided, comprising the following steps:

[0061] Step 101, during the user's vocalization process, obtain the user's lip shape information and the voice information of the target sound emitted.

[0062] Here, during the user's vocalization process, the user's lip shape information and voice information of the emitted target sound can be obtained through the electronic device.

[0063] In a possible embodiment, during the user's vocalization, the user's lip shape information and the voice information of the target sound emitted can be obtained respectively through an image acquisition device and an audio acquisition device. It should be noted that the selection of the image acquisition device and the audio acquisition device is no longer limited here, as long as the corresponding acquisition function can be realized.

[0064] Step 102, determining the backup initial consonant information of each word in the target sound based on the lip shape information.

[0065] Here, after acquiring the user's lip shape information and the voice information of the target sound, the electronic device can determine the backup initial consonant information of each word in the target sound based on the lip shape information.

[0066] In a possible embodiment, determining the spare initial consonant information of each word in the target sound based on the lip shape information includes the following steps:

[0067] The lip shape information is input into the improved YOLOv8 model for processing to obtain the backup initial consonant information of each word in the target sound.

[0068] Specifically, after obtaining the user's lip shape information and the voice information of the target sound, the electronic device can input the lip shape information into the improved YOLOv8 model for processing to obtain the backup initial consonant information of each word in the target sound. Figure 2 As shown, Figure 2 The structural diagram of the improved YOLOv8 model is exemplarily shown. The improved YOLOv8 model includes a backbone network, a neck module and a head module. The model adds an EMA attention mechanism module and an MSAM multi-scale convolution attention module on the basis of the original YOLOv8 model. Among them, the backbone network of the improved YOLOv8 model is composed of a Conv convolution module and a C2f module. The connection mode of the backbone network can be: Conv-Conv-C2f-Conv-C2f-Conv-C2f-Conv-C2f. It should be noted that the backbone network can also be For other connection methods, the specific connection method of the backbone network is not limited here, and its functions shall be realized; the neck module of the improved YOLOv8 model is composed of an SPPF network, an Upsample module, a Concat module, a C2f module and a Conv module. Similarly, the neck module can also be composed of other modules. The specific module types in the neck module and the connection methods between modules are not limited here; the improved YOLOv8 model is composed of three Detest modules. Similarly, the head module can also be composed of other modules, which is not limited here, and its functions shall be realized.

[0069] In this embodiment, the added EMA attention mechanism module is set between the backbone network and the neck module of the improved YOLOv8 model. The EMA attention mechanism module can divide the input lip shape information into G sub-features for learning different semantics. Without losing generality, the sub-feature range is set, and it is assumed that the learned attention weight descriptor will be used to enhance the feature representation of the region of interest in each sub-feature, where G<<C, C represents the number of channels of the attention mechanism module. The added EMA attention mechanism module can extract the attention weight descriptor of the grouped feature map composed of lip shape information through three parallel routes. Two parallel paths are on the 1x1 branch, and the third path is on the 3x3 branch. In order to capture the dependencies between all channels and reduce the computational budget, two global average pooling operations are used to encode the channels along two spatial directions in the 1x1 branch, and only one 3x3 kernel is stacked in the 3x3 branch to capture multi-scale features.

[0070] In addition, the EMA attention mechanism module proposes a cross-spatial information aggregation method in different spatial dimensions to achieve richer feature aggregation. The output of the 1x1 branch is encoded with global spatial information using two-dimensional global average pooling, and the two-dimensional global pooling operation formula can be further obtained. The formula is as follows:

[0071]

[0072] Among them, Z c represents the aggregated feature map of the cth channel, x c represents the pixel of the cth channel, i and j represent integers, and H and W are the height and width of the lip shape feature map.

[0073] In this embodiment, the added MSAM multi-scale convolutional attention module is set after the SPPF network to make the channel attention multi-scale. In the traditional YOLOv8 model, CBMA (Convolutional Block Attention Module) is usually integrated after the convolution layer to introduce spatial attention and channel attention to help the model better focus on the key areas and channels in the image. In this embodiment, another multi-scale convolutional attention module MSCA is introduced to make the channel attention multi-scale. MSCA consists of three parts: deep convolution aggregates local information, multi-branch deep strip convolution captures multi-scale context, and 1×1 convolution models the relationship between different channels. After introducing MSCA into CMBA, the MSAM multi-scale convolutional attention module can be obtained. The MS AM multi-scale convolutional attention module is used to find the side areas that contribute most to the final HR reconstruction (high-resolution reconstruction) and give them greater weights.

[0074] The electronic device inputs the lip shape information into the improved YOLOv8 model, and the lip shape information is processed by the backbone network, neck module and head module in the improved YOLOv8 model in turn, and finally outputs the backup initial consonant information of each word in the target sound in the Detect module of the head module. It should be noted that the initial sample set of the YOLOv8 model is the lip shape information obtained by the image acquisition device within a preset time when the user turns on the image acquisition device for the first time. As the lip shape information obtained by the YOLOv8 model continues to increase, when the lip shape information reaches a preset threshold, the electronic device can update the sample set of the YOLOv8 model to make the model more in line with the user's habits.

[0075] Step 103: perform speech recognition on the speech information, obtain text information after the recognition, and determine at least one target sensitive word in the text information.

[0076] Here, after the electronic device determines the backup initial consonant information of each word in the target sound based on the lip shape information, it can perform speech recognition on the speech information, obtain text information after recognition, and determine at least one target sensitive word in the text information.

[0077] In a possible embodiment, performing speech recognition on speech information to obtain text information after recognition, and determining at least one target sensitive word in the text information includes the following steps:

[0078] Input the speech information into a pre-built speech recognition model to obtain text information;

[0079] At least one target sensitive word is identified in the text information through a decision tree.

[0080] Specifically, after the electronic device determines the backup initial consonant information of each word in the target sound based on the lip shape information, the voice information is input into a pre-built voice recognition model to obtain text information, wherein the method for building the voice recognition model is: the electronic device establishes a voice recognition model based on a natural language processing model, and according to the natural language processing theory, counts a series of voice information input by the user, and represents the collection with Q, and the specific expression is shown in the following formula:

[0081] Q={t1,t2...t n ,}.

[0082] Among them, t1, t2...t n Represents the acquired voice information, n represents the voice information sequence number. T is used to represent the frequency of input voice information in set Q. The specific expression is as follows:

[0083] T={(c i1 ,f),(c i2 ,f)...(c is ,f)}.

[0084] Among them, c i1 ,c i2 ,c is Respectively represent the definition of different voice information input, is represents the sequence number, and f represents the definition frequency. In this way, the initial establishment of the voice recognition model is completed. At this time, the input content of the voice recognition model is voice information, and the output content is the text information of the voice information. It should be noted that as the voice information input by the user continues to increase, when the amount of input voice information reaches a preset threshold, the voice recognition model can be updated, so that the voice recognition model can be more in line with the user's usage habits.

[0085] In an optional embodiment, in the process of constructing a speech recognition model, the original waveform of the speech information can be first obtained, and frame processing can be performed according to the original waveform. For example, each low-frequency signal is used as a frame end signal to obtain multiple frame signals, and the frame signal is used as an initial feature parameter, and then the initial feature parameter is subjected to noise reduction processing, wherein the noise reduction processing process for the initial feature parameter includes: according to the short-term stability principle, the time range of the frame length is fixed within a preset time range, for example, the preset time range can be 10-30ms, and all frame signals are processed by an overlapping frame method to obtain a target frame signal, wherein, in the overlapping frame processing process, the overlapping part of the previous and next frames is represented by a frame shift. The frame signal is processed by overlapping frame division to avoid errors at both ends of the signal, so that when the signal is recognized, the generated speech recognition model can obtain text information more accurately. After the initial feature parameter is subjected to noise reduction processing, the electronic device regards each frame signal as an independent random signal, and adopts discrete cosine transform to obtain noise reduction feature parameters, and then, the preset natural language processing model converts the input speech information into corresponding text information through the speech recognition model according to the noise reduction feature parameters.

[0086] Through the above-mentioned embodiment, the electronic device inputs the voice information into the pre-built voice recognition model, and after obtaining the text information, at least one target sensitive word can be identified in the text information through the decision tree.

[0087] In a possible embodiment, identifying at least one target sensitive word in text information through a decision tree includes the following steps:

[0088] Determine the appearance position and frequency of each text message in the voice message;

[0089] The occurrence position and frequency are input into the decision tree for processing to obtain the word sensitivity of the text information;

[0090] When the word sensitivity of the text information is greater than a preset sensitivity threshold, the text information is determined to be a target sensitive word.

[0091] Specifically, the electronic device inputs the voice information into a pre-built voice recognition model. After obtaining the text information, first, the appearance position and frequency of each text information in the voice information are determined. Then, the electronic device inputs the appearance position and frequency into a decision tree for processing to obtain the word sensitivity of the text information. Each node in the decision tree stores information such as sensitive words, sensitive word frequency, and sensitivity level. The appearance position and frequency of each text information in the voice information are input into a preset decision tree, and the word sensitivity of the text information can be obtained. After obtaining the word sensitivity of the text information, it can be determined whether the text information is a target sensitive word according to the word sensitivity of the text information and a first preset sensitivity threshold. When the word sensitivity of the text information is greater than the first preset sensitivity threshold, the text information is determined to be a target sensitive word. The definition set D of the determined target sensitive word can be expressed as:

[0092] D = {w i1 ,w i2 ,...,w ij ,...,w im}.

[0093] Among them, w i1 ,w i2 ,...,w ij ,...,w im They represent different target sensitive words respectively.

[0094] In an optional embodiment, after determining the target sensitive word, the electronic device may obtain a sensitivity calculation result according to the length of the text information output by the speech recognition model and the error coefficient of the target sensitive word in order to balance the influence of the speech length on the intelligent recognition of the sensitive word, and screen the target sensitive word according to the sensitivity calculation result. The calculation formula of the sensitivity calculation result is:

[0095] k=len×s.

[0096] Among them, the text length and the error coefficient of the target sensitive word are represented by len and s respectively, and k is the sensitivity calculation result. It should be noted that the error coefficient s of the target sensitive word can be user-defined or generated based on historical big data, and there is no restriction here.

[0097] After calculating the sensitivity calculation result of the target sensitive word, the target sensitive word can be further processed according to the second preset sensitivity threshold and the sensitivity calculation result. Specifically, when the sensitivity calculation result of the target sensitive word is greater than the second preset sensitivity threshold, the target sensitive word is retained, and when the sensitivity calculation result of the target sensitive word is less than the second preset sensitivity threshold, the target sensitive word is removed. Exemplarily, the second preset sensitivity threshold can be set to 0, and the target sensitive words determined by the electronic device are A and B, wherein the sensitivity calculation result of the target sensitive word A is 1, and the sensitivity calculation result of the target sensitive word B is -3, and the sensitivity calculation result of the target sensitive word A is greater than the second preset sensitivity threshold 0, then the target sensitive word A is retained, and the sensitivity calculation result of the target sensitive word A is less than the second preset sensitivity threshold 0, then the target sensitive word B is removed.

[0098] Step 104, based on the backup initial consonant information and the target sensitive word, determine the speech recognition result of the target sound, and determine the task to be performed based on the speech recognition result.

[0099] Here, after the electronic device determines at least one target sensitive word in the text information, it can determine the speech recognition result of the target sound based on the backup initial consonant information and the target sensitive word, and determine the task to be performed based on the speech recognition result.

[0100] In a possible embodiment, determining a speech recognition result of a target sound based on the backup initial consonant information and the target sensitive word includes the following steps:

[0101] When the initial consonant set includes the initial consonants of each sensitive character in the target sensitive word, determining the speech recognition result based on the target sensitive word;

[0102] In the case that the initial consonant set does not include at least some of the initial consonants of the sensitive characters, the speech recognition result is determined based on the initial consonant set.

[0103] Specifically, after the electronic device determines at least one target sensitive word in the text information, the electronic device calculates the matching degree between the backup initial consonant information and the target sensitive word, and the calculation formula is:

[0104]

[0105] Wherein, v represents the initial consonant of each sensitive word in the target sensitive word, Φ represents the initial consonant set, the initial consonant set includes the spare initial consonant information determined by the lip shape information, FLAG represents the flag bit, when the initial consonant set includes the initial consonants of each sensitive word in the target sensitive word, FLAG=1, at this time, the speech recognition result can be determined based on the target sensitive word, and the target sensitive word is composed of the target text information, and the target text information is the speech recognition result; when the initial consonant set does not include the initial consonants of at least some sensitive words, FLAG=0, at this time, the speech recognition result can be determined based on the initial consonant set, the initial consonant set can be input into the decision tree, the initial consonant set is processed by the decision tree, the target sensitive word composed of the initial consonant set is obtained, and the target sensitive word is composed of the target text information, and the target text information is output as the speech recognition result. After the electronic device determines the speech recognition result of the target sound according to the calculation result of the matching degree between the spare initial consonant information and the target sensitive word, the electronic device determines the task to be performed based on the speech recognition result.

[0106] The embodiments of the present disclosure provide a speech recognition method, apparatus, device, medium and product. In the embodiments of the present application, first, during the user's speech, the user's lip shape information and the speech information of the target sound emitted are obtained; then, the backup initial consonant information of each word in the target sound is determined based on the lip shape information; thereafter, speech recognition is performed on the speech information, and text information is obtained after recognition, and at least one target sensitive word is determined in the text information; finally, the speech recognition result of the target sound is determined based on the backup initial consonant information and the target sensitive word, and the task to be performed is determined based on the speech recognition result.

[0107] From the above description, it can be seen that the present disclosure obtains the user's lip shape information and voice information respectively during the user's voice utterance, and determines the backup initial consonant information and target sensitive words based on the lip shape information and voice information respectively, and finally determines the voice recognition result through the backup initial consonant information and the target sensitive word, thereby determining the task to be executed. The present disclosure realizes the method of combining the user's lip shape information and voice information to determine the voice recognition result. Compared with the traditional technology of only performing voice recognition on the user's voice, the voice recognition result determined by the present disclosure is more accurate and reliable.

[0108] In one embodiment, Figure 3 As shown, in step 102, the lip shape information is input into the improved YOLOv8 model for processing to obtain the spare initial consonant information of each word in the target sound, and the following steps are also included:

[0109] Step 301, extracting features of lip shape information through the backbone network in the YOLOv8 model to obtain a lip shape feature map.

[0110] Here, after obtaining the user's lip shape information and the voice information of the target sound emitted, the electronic device can extract features of the lip shape information through the backbone network in the YOLOv8 model to obtain a lip shape feature map.

[0111] In a possible embodiment, Figure 2 As shown, after obtaining the user's lip shape information and the voice information of the target sound, the electronic device inputs the user's lip shape information into the backbone network in the YOLOv8 model, and extracts the lip shape information through the backbone network in the YOLOv8 model to obtain a lip shape feature map. It should be noted that in this embodiment, Figure 2 The backbone network in the structural diagram of the improved YOLOv8 model shown is only an exemplary display of the structure of one of the backbone networks. The backbone network can also be other structures, which is not limited here.

[0112] The lip shape feature map obtained by the backbone network in the YOLOv8 model can be expressed by the following formula:

[0113] X∈R C*H*W .

[0114] Among them, X represents the lip feature map, H represents the height of the lip feature map, W represents the width of the lip, C represents the number of channels of the lip feature map, and R represents the set of lip feature maps.

[0115] Step 302: Input the lip shape feature map into the EMA attention mechanism module for processing to obtain a first attention matrix.

[0116] Here, after the electronic device extracts the features of the lip shape information through the backbone network in the YOLOv8 model and obtains a lip shape feature map, the lip shape feature map can be input into the EMA attention mechanism module for processing to obtain a first attention matrix.

[0117] In a possible embodiment, Figure 2 As shown, after the electronic device extracts the lip shape information through the backbone network in the YOLOv8 model to obtain the lip shape feature map, the lip shape feature map is input into the EMA attention mechanism module located between the backbone network and the SPPF network for processing to obtain the first attention matrix, wherein the process of processing the lip shape feature map by the EMA attention mechanism includes the following steps:

[0118] Step 1: Group the input lip shape feature map. The channels of the input feature map are divided into multiple groups to reduce the amount of calculation. Assume that the shape of the input feature map is C×H×W, where C is the number of channels, H and W are the height and width of the lip shape feature map. Through grouping, the lip shape feature map is divided into G groups, and the size of each group of lip shape feature maps is C / G×H×W;

[0119] Step 2: Perform global pooling in the X and Y directions on each set of lip shape feature maps, and perform average pooling in the X and Y directions on each set of lip shape feature maps to obtain two feature maps, representing the global features in the X and Y directions respectively;

[0120] Step 3: Concatenate the two feature maps. After concatenating the two feature maps, perform feature fusion through 1×1 convolution and output a new feature map with a shape of C / G×H×W to further capture cross-space information.

[0121] Step 4: Generate a weight coefficient through the Sigmoid function, and then re-weight the new feature map to obtain a new lip shape feature map to enhance the area with important features and suppress the unimportant areas;

[0122] Step 5: Normalize each set of new lip shape feature maps, keep each set of features at the same scale, generate a cross-space attention matrix through global average pooling and softmax function, and then perform interactive learning through matrix multiplication. Finally, the weighted lip shape feature map is output, that is, the first attention matrix, and the shape is restored to C×H×W.

[0123] Step 303, perform feature map fusion processing on the first attention matrix through the neck module in the YOLOv8 model, and detect the fusion result through the head module in the YOLOv8 model to obtain the backup initial consonant information of each word in the target sound.

[0124] Here, after the electronic device inputs the lip shape feature map into the EMA attention mechanism module for processing and obtains the first attention matrix, it can perform feature map fusion processing on the first attention matrix through the neck module in the YOLOv8 model, and detect the fusion result through the head module in the YOLOv8 model to obtain the backup initial consonant information of each word in the target sound.

[0125] In a possible embodiment, the feature map fusion processing is performed on the first attention matrix through the neck module in the YOLOv8 model, including the following steps:

[0126] Input the first attention matrix into the SPPF network in the neck module for processing to obtain a first processing result;

[0127] Processing the first processing result through the MSAM multi-scale convolution attention module to obtain a second processing result;

[0128] The second processing result is subjected to feature fusion processing through the remaining network layers in the neck module to obtain a fusion result.

[0129] Specifically, Figure 2As shown, first, the electronic device inputs the lip shape feature map into the EMA attention mechanism module for processing to obtain a first attention matrix, and then inputs the first attention matrix into the SPPF network in the neck module for processing to obtain a first processing result; then, the electronic device inputs the first processing result into the MSAM multi-scale convolutional attention module in the YOLOv8 model, and the MSAM multi-scale convolutional attention module is located in the neck model of the YOLOv8 model, and is arranged between the SPFF network and the Upsample module, and the MSAM multi-scale convolutional attention module is used to perform the final HR reconstruction (high-resolution reconstruction) on the first processing result and output the second processing result; finally, the electronic device performs feature fusion processing on the second processing result through the remaining network layers in the neck module, namely, the network layers composed of the Upsample module, the Concat module, the C2f module and the Conv module, and finally obtains the fusion result.

[0130] After obtaining the fusion result, the electronic device can detect the fusion result through the head module in the YOLOv8 model to obtain the backup initial consonant information of each word in the target sound.

[0131] In a possible embodiment, Figure 2 As shown, after obtaining the fusion result, the electronic device inputs the fusion result into the head module in the YOLOv8 model. The head module is composed of three Detest modules. The head module can detect the fusion result and finally output the spare initial consonant information of each word in the target sound. The spare initial consonant information of each word outputted can be one initial consonant or multiple initial consonants. The number of output initial consonants is not limited here. It should be noted that in this embodiment, Figure 2 The head module in the structural diagram of the improved YOLOv8 model shown is only an exemplary illustration of the structure of one of the head modules. The head module may also have other structures, which are not limited here.

[0132] In this embodiment, first, the electronic device performs feature extraction on the lip shape information through the backbone network in the YOLOv8 model to obtain a lip shape feature map. Then, the electronic device inputs the lip shape feature map into the EMA attention mechanism module for processing to obtain a first attention matrix. Finally, the electronic device performs feature map fusion processing on the first attention matrix through the neck module in the YOLOv8 model, and detects the fusion result through the head module in the YOLOv8 model to obtain the backup initial consonant information of each word in the target sound.

[0133] From the above description, it can be seen that the electronic device inputs the lip shape feature map into the EMA attention mechanism module for processing to obtain the first attention matrix, wherein the EMA attention mechanism module is added to the improved YOLOv8 model, and the EMA attention mechanism module can retain the information of each channel by reshaping some channels to the batch dimension and grouping the channel dimension into multiple sub-features, so that the features of the obtained first attention matrix are more accurate; and the electronic device performs feature map fusion processing on the first attention matrix through the neck module in the YOLOv8 model, and the neck module in the YOLOv8 model adds the MSAM multi-scale convolution attention module, which can not only improve the fusion accuracy of the first attention matrix, but also improve the fusion result under background interference. Finally, under the joint action of the EMA attention mechanism module and the MSAM multi-scale convolution attention module, the backup initial consonant information of each word in the obtained target sound is made more accurate, further improving the reliability of the speech recognition method.

[0134] In one embodiment, Figure 4 As shown, in step 104, determining the task to be performed based on the speech recognition result also includes the following steps:

[0135] Step 401, matching the speech recognition result with the speech task in the speech task library to obtain the matching degree between the speech task and the speech recognition result.

[0136] In a possible embodiment, after determining the speech recognition result of the target sound, the electronic device may match the speech recognition result with the speech tasks in the speech task library to obtain a matching degree between the speech tasks and the speech recognition result.

[0137] Step 402: Determine the voice tasks whose matching degree meets the preset conditions as tasks to be executed.

[0138] In a possible embodiment, after obtaining the matching degree between the voice task and the voice recognition result, the electronic device determines the voice task whose matching degree meets the preset conditions as the task to be executed. For example, the matching degrees between task A and task B in the voice task library and the voice recognition result are 95% and 86% respectively, and the electronic device determines task A with a greater matching degree as the task to be executed.

[0139] In another possible embodiment, after obtaining the matching degree between the voice task and the voice recognition result, if the matching degree of all tasks in the voice task library is less than a preset condition, the electronic device issues a correction prompt to the user. For example, if the matching degree between all tasks in the voice task library and the voice recognition result is less than 80%, the electronic device may issue a prompt message to the user in the form of voice, "The task you posted may be wrong, please post it again." It should be noted that the form and content of the correction prompt issued by the electronic device are not limited here.

[0140] In this embodiment, first, the electronic device matches the speech recognition result with the speech task in the speech task library to obtain the matching degree between the speech task and the speech recognition result; then, the electronic device determines the speech task whose matching degree meets the preset conditions as the task to be executed.

[0141] From the above description, it can be seen that the electronic device can determine the task to be executed based on the degree of matching between the voice task and the voice recognition result, and can issue an error correction prompt to the user when the matching degree of all tasks in the voice task library is less than the preset condition, thereby improving the humanization of the voice recognition method and thus improving the user's interactive experience.

[0142] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a speech recognition device, which may be a server or a chip applied to a server. Figure 5 The following is a schematic block diagram of the functional modules of a speech recognition device provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, the speech recognition device comprises:

[0143] The acquisition module 501 is used to acquire the lip shape information of the user and the voice information of the target sound emitted by the user during the user's sound production process;

[0144] A first determination module 502, configured to determine the spare initial consonant information of each word in the target sound based on the lip shape information;

[0145] The second determination module 503 is used to perform speech recognition on the speech information, obtain text information after recognition, and determine at least one target sensitive word in the text information;

[0146] The third determination module 504 is used to determine the speech recognition result of the target sound based on the backup initial consonant information and the target sensitive word, and determine the task to be performed based on the speech recognition result.

[0147] In one embodiment, the first determining module 502 includes:

[0148] The spare initial consonant information determining unit is used to input the lip shape information into the improved YOLOv8 model for processing to obtain the spare initial consonant information of each word in the target sound.

[0149] In one embodiment, the first determining module 502 includes:

[0150] A first acquisition unit is used to extract features of the lip shape information through a backbone network in the YOLOv8 model to obtain a lip shape feature map;

[0151] A second acquisition unit is used to input the lip shape feature map into the EMA attention mechanism module for processing to obtain a first attention matrix;

[0152] The third acquisition unit is used to perform feature map fusion processing on the first attention matrix through the neck module in the YOLOv8 model, and detect the fusion result through the head module in the YOLOv8 model to obtain the backup initial consonant information of each word in the target sound.

[0153] In one embodiment, the first determining module 502 includes:

[0154] A fourth acquisition unit, used for inputting the first attention matrix into the SPPF network in the neck module for processing to obtain a first processing result;

[0155] a fifth acquisition unit, configured to process the first processing result through a MSAM multi-scale convolutional attention module to obtain a second processing result;

[0156] The sixth acquisition unit is used to perform feature fusion processing on the second processing result through the remaining network layers in the neck module to obtain a fusion result.

[0157] In one embodiment, the second determining module 503 includes:

[0158] a seventh acquisition unit, configured to input the voice information into a pre-built voice recognition model to obtain the text information;

[0159] An eighth acquisition unit is configured to identify at least one target sensitive word in the text information through a decision tree.

[0160] In one embodiment, the second determining module 503 includes:

[0161] A first determining unit, used to determine the occurrence position and frequency of each of the text information in the voice information;

[0162] a ninth acquisition unit, configured to input the occurrence position and the occurrence frequency into the decision tree for processing to obtain the word sensitivity of the text information;

[0163] The second determining unit is configured to determine that the text information is the target sensitive word if the word sensitivity of the text information is greater than a preset sensitivity threshold.

[0164] In one embodiment, the third determining module 504 includes:

[0165] A third determining unit, configured to determine the speech recognition result based on the target sensitive word when the initial consonant set includes the initial consonants of each sensitive word in the target sensitive word;

[0166] The fourth determination unit is used to determine the speech recognition result based on the initial consonant set when the initial consonant set does not include at least some of the initial consonants of the sensitive word; wherein the initial consonant set includes the backup initial consonant information.

[0167] In one embodiment, the third determining module 504 includes:

[0168] A matching unit, used for matching the speech recognition result with the speech task in the speech task library to obtain a matching degree between the speech task and the speech recognition result;

[0169] The fifth determining unit is used to determine the voice task whose matching degree meets the preset condition as the task to be executed.

[0170] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0171] Figure 6 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the electronic device 600 includes at least one processor 601 and a memory 602 coupled to the processor 601 , and the processor 601 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0172] The processor 601 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 601 or by instructions in the form of software. The processor 601 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 602, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 601 reads the information in the memory 602 and completes the steps of the method in combination with its hardware.

[0173] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 7 The computer system 700 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 7 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0174] Computer system 700 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0175] like Figure 7As shown, the computer system 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the computer system 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0176] A plurality of components in the computer system 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the computer system 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include, but is not limited to, a disk, an optical disk. The communication unit 709 allows the computer system 700 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0177] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 702 and / or the communication unit 709. In some embodiments, the computing unit 701 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0178] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0179] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0180] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0181] Figure 8 A computer program product is provided for an exemplary embodiment of the present disclosure. The computer program product 800 includes a computer program 801. When the computer program 801 is executed by a processor, the method disclosed in the embodiment of the present disclosure is implemented.

[0182] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0183] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0184] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0185] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0186] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0187] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A speech recognition method, characterized in that: include: During the user's utterance, obtaining the user's lip shape information and the voice information of the target sound emitted; Determine the backup initial consonant information of each word in the target sound based on the lip shape information; Performing speech recognition on the speech information to obtain text information after the recognition, and determining at least one target sensitive word in the text information; Based on the backup initial consonant information and the target sensitive word, a speech recognition result of the target sound is determined, and based on the speech recognition result, a task to be performed is determined.

2. The method according to claim 1, characterized in that The step of determining the spare initial consonant information of each word in the target sound based on the lip shape information comprises: The lip shape information is input into the improved YOLOv8 model for processing to obtain the backup initial consonant information of each word in the target sound.

3. The method according to claim 2, characterized in that The step of inputting the lip shape information into the improved YOLOv8 model for processing to obtain the spare initial consonant information of each word in the target sound includes: Performing feature extraction on the lip shape information through the backbone network in the YOLOv8 model to obtain a lip shape feature map; Input the lip shape feature map into the EMA attention mechanism module for processing to obtain a first attention matrix; The first attention matrix is ​​subjected to feature map fusion processing through the neck module in the YOLOv8 model, and the fusion result is detected through the head module in the YOLOv8 model to obtain the backup initial consonant information of each word in the target sound.

4. The method according to claim 3, characterized in that The performing feature map fusion processing on the first attention matrix through the neck module in the YOLOv8 model includes: Inputting the first attention matrix into the SPPF network in the neck module for processing to obtain a first processing result; Processing the first processing result through a MSAM multi-scale convolutional attention module to obtain a second processing result; The second processing result is subjected to feature fusion processing through the remaining network layers in the neck module to obtain a fusion result.

5. The method according to claim 1, characterized in that The performing speech recognition on the speech information to obtain text information after the recognition, and determining at least one target sensitive word in the text information, includes: Inputting the voice information into a pre-built voice recognition model to obtain the text information; At least one target sensitive word is identified in the text information through a decision tree.

6. The method according to claim 5, characterized in that The step of identifying at least one target sensitive word in the text information through a decision tree includes: Determine the occurrence position and frequency of each of the text information in the voice information; Inputting the occurrence position and the occurrence frequency into the decision tree for processing to obtain the word sensitivity of the text information; When the word sensitivity of the text information is greater than a preset sensitivity threshold, the text information is determined to be the target sensitive word.

7. The method according to claim 1, characterized in that The determining the speech recognition result of the target sound based on the backup initial consonant information and the target sensitive word includes: When the initial consonant set includes the initial consonants of each sensitive word in the target sensitive word, determining the speech recognition result based on the target sensitive word; In the case that the initial consonant set does not include at least some of the initial consonants of the sensitive characters, a speech recognition result is determined based on the initial consonant set; wherein the initial consonant set includes the backup initial consonant information.

8. The method according to claim 1, characterized in that The determining of the task to be performed based on the speech recognition result includes: Matching the speech recognition result with the speech tasks in the speech task library to obtain a matching degree between the speech task and the speech recognition result; The voice task whose matching degree meets a preset condition is determined as the task to be executed.

9. A speech recognition device, characterized in that: include: An acquisition module, used to acquire the lip shape information of the user and the voice information of the target sound emitted during the user's utterance; A first determination module, used for determining the spare initial consonant information of each word in the target sound based on the lip shape information; A second determination module is used to perform speech recognition on the speech information, obtain text information after recognition, and determine at least one target sensitive word in the text information; The third determination module is used to determine the speech recognition result of the target sound based on the backup initial consonant information and the target sensitive word, and determine the task to be performed based on the speech recognition result.

10. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images