Subtitle matching display method and system based on audio file processing

Abstract

The invention provides a subtitle matching display method and system based on audio file processing, and relates to the technical field of audio processing, and the method comprises the steps: determining a target region where a target person is located, determining the demand coefficient of Internet of Things equipment according to the historical starting time of the Internet of Things equipment in the target region, and then determining the target Internet of Things equipment; determining an audio file according to the type and the function of the target Internet of Things device; determining a function menu according to the audio file and the type and the function of the target Internet of Things equipment; generating recommendation setting information according to the audio file, the type and the function of the target Internet of Things equipment and a selection button in a function menu; and according to the recommendation setting information and the audio file, generating recommendation text information, and further generating subtitles. According to the method and the device, the subtitles can be generated when the Internet of Things equipment plays the audio file, so that the operation prompt is performed on the target personnel, the target personnel can correctly operate the Internet of Things equipment, and the use convenience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to a subtitle matching display method and system based on audio file processing. Background Art

[0002] In the related art, CN106504754B discloses a method for generating real-time subtitles based on audio output, and the steps are as follows: for the audio information that the electronic device needs to output, the following operations are performed: the audio information output by the electronic device is monitored in real time by an audio acquisition module, and collected; the collected audio information is transmitted to a voice extraction module, and irrelevant content such as background music in the audio information is filtered and noise reduction is performed to obtain accurate voice information; thereafter, the obtained voice information that needs to be converted into text is input into a voice recognition module to obtain text information corresponding to the voice; finally, the converted text is displayed in real time on the device screen in the form of subtitles by a display module. The advantages of this solution are: it can help the hearing-impaired obtain the voice content contained in video, audio or other forms, and provides an effective and convenient way for the hearing-impaired to obtain voice information, while also providing convenience for ordinary users.

[0003] CN117809654B discloses a low-resource audio subtitle generation method, device, electronic device and medium. By applying this technical solution, in a multimodal pre-training model including a language encoder and an audio encoder, text data with a relatively sufficient sample size can be first used to train an existing language encoder to obtain a language decoder. Subsequently, the language encoder is replaced with an audio encoder to achieve the purpose of indirectly training the audio encoder to obtain a language decoder. In this way, a high-precision audio multimodal pre-training model can be trained using only audio pairing data with a small sample size. Thereby, a technical solution that can still achieve high model performance when there is only a small amount of audio-subtitle data pairs available for training is realized.

[0004] Therefore, using relevant technologies, the content in the audio can be converted into text to generate subtitles, but the relevant technologies can only make the subtitles consistent with the content of the audio, but cannot explain the content of the audio in more detail, nor can they serve as a reminder to the viewers.

[0005] The information disclosed in the background technology section of this application is only intended to deepen the understanding of the general background technology of this application, and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0006] The present invention provides a subtitle matching display method and system based on audio file processing, which can solve the technical problem that the related art cannot explain the content of audio in more detail.

[0007] According to a first aspect of the present invention, there is provided a subtitle matching and display method based on audio file processing, comprising:

[0008] Determine a target area where a target person is located in a plurality of areas by using an infrared sensor disposed on an IoT device, wherein each area includes at least one IoT device, and the IoT device includes a screen and an audio component;

[0009] Determine the target person’s demand coefficient for multiple IoT devices based on the historical startup time of IoT devices in the target area;

[0010] Determining target IoT devices for launching screen and audio components based on the demand coefficients;

[0011] Determine the audio files to be played through the audio component based on the type and capabilities of the target IoT device;

[0012] Determine a function menu displayed on the screen according to the audio file and the type and function of the target IoT device;

[0013] Generate recommended setting information according to the audio file, the type and function of the target IoT device, and the selection button in the function menu;

[0014] generating recommended text information according to the recommended setting information and the audio file;

[0015] According to the recommended text information, subtitles displayed in the function menu are determined.

[0016] According to a second aspect of the present invention, there is provided a subtitle matching and display system based on audio file processing, comprising:

[0017] A target area module, used to determine the target area where the target person is located in multiple areas by using an infrared sensor set on the IoT device, wherein each area includes at least one IoT device, and the IoT device includes a screen and an audio component;

[0018] A demand coefficient module is used to determine the demand coefficients of the target personnel for multiple IoT devices based on the historical startup times of the IoT devices in the target area;

[0019] A target IoT device module, used for determining a target IoT device for starting a screen and audio component according to the demand coefficient;

[0020] An audio file module, used to determine the audio file to be played through the audio component based on the type and function of the target IoT device;

[0021] A function menu module, used to determine a function menu displayed on the screen according to the audio file and the type and function of the target IoT device;

[0022] A recommended setting information module, used to generate recommended setting information according to the audio file, the type and function of the target IoT device, and the selection button in the function menu;

[0023] A recommended text information module, used to generate recommended text information according to the recommended setting information and the audio file;

[0024] The subtitle module is used to determine the subtitles displayed in the function menu according to the recommended text information.

[0025] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0026] According to the present invention, the target person's demand coefficient for multiple IoT devices can be determined by the target person's historical startup time for the IoT device, thereby determining the IoT device to be started, and when the IoT device plays an audio file to prompt the target person to operate, subtitles can be generated, thereby providing operation prompts to the target person, so that the target person can correctly operate the IoT device and improve the convenience of use. When determining the demand coefficient, the target person's demand for the IoT device can be comprehensively described by the probability of starting the IoT device after entering the target area and the urgency of starting the IoT device, thereby obtaining the demand coefficient of the IoT device, and improving the comprehensiveness and accuracy of the demand coefficient. When determining the startup demand coefficient, the average relative deviation of the environmental parameters corresponding to various functions and the maximum value of the average relative change can be used to describe the overall demand of the current environment for starting various functions, thereby improving the objectivity and accuracy of the startup demand coefficient. When training a text generation model, a conditional function can be used to determine whether the overall semantics of the sample recommendation text is wrong, and the different aspects of the text generation model can be improved when there is no error and when there is an error, that is, the text generation model's ability to polish the text and the accuracy of the description of the text are improved, thereby improving the accuracy of the text generation model and improving the training efficiency.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present invention. Other features and aspects of the present invention will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative work.

[0029] Figure 1 A schematic flow chart of a subtitle matching and display method based on audio file processing according to an embodiment of the present invention is exemplarily shown;

[0030] Figure 2 A block diagram of a subtitle matching and display system based on audio file processing according to an embodiment of the present invention is exemplarily shown. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0033] Figure 1 A schematic flow chart of a subtitle matching and display method based on audio file processing according to an embodiment of the present invention is exemplarily shown, and the method includes:

[0034] Step S101, determining a target area where a target person is located in a plurality of areas by using an infrared sensor provided on an IoT device, wherein each area includes at least one IoT device, and the IoT device includes a screen and an audio component;

[0035] Step S102, determining the demand coefficient of the target person for multiple IoT devices according to the historical startup time of the IoT devices in the target area;

[0036] Step S103, determining a target IoT device for starting screen and audio components according to the demand coefficient;

[0037] Step S104, determining the audio file to be played through the audio component according to the type and function of the target IoT device;

[0038] Step S105, determining a function menu displayed on the screen according to the audio file and the type and function of the target IoT device;

[0039] Step S106, generating recommended setting information according to the audio file, the type and function of the target IoT device, and the selection button in the function menu;

[0040] Step S107, generating recommended text information according to the recommended setting information and the audio file;

[0041] Step S108: determining subtitles to be displayed in the function menu according to the recommended text information.

[0042] According to the subtitle matching display method based on audio file processing of an embodiment of the present invention, the target person's demand coefficient for multiple IoT devices can be determined through the target person's historical startup time for the IoT device, thereby determining the IoT device to be started, and when the IoT device plays an audio file to prompt the target person to operate, subtitles can be generated, thereby providing operation prompts to the target person, so that the target person can correctly operate the IoT device and improve ease of use.

[0043] According to one embodiment of the present invention, in step S101, the IoT device may include household appliances and other devices, such as air conditioners, stereos, humidifiers, etc. The area may be a room, each area may include at least one IoT device, and among one or more IoT devices in the same area, at least one IoT device includes an infrared sensor, which can detect whether the target person is in the area. If the target person is in the area, the area is determined as the target area. In addition, the IoT device may include a screen and an audio component. The screen can display a specific screen and can also receive touch input from the target person. The audio component may include a sound playing component (such as a stereo) for playing audio files, and a sound receiving component (such as a microphone) for receiving sound signals. For example, the sound receiving component can receive the voice input command of the target person and can also be used to detect noise in the environment.

[0044] According to one embodiment of the present invention, in step S102, the demand coefficient can be used to represent the target person's demand for the use of the Internet of Things devices in the target area. According to the historical startup time of the Internet of Things devices in the target area, the demand coefficient of the target person for multiple Internet of Things devices is determined, including: determining multiple target time periods in which the target person is in the target area in multiple historical periods; determining the historical startup time of each Internet of Things device in each target time period; determining the startup sequence of each Internet of Things device in each target time period according to the historical startup time of each Internet of Things device in each target time period; determining whether each Internet of Things device is started in each target time period; determining the demand coefficient of each Internet of Things device according to the startup sequence of each Internet of Things device in each target time period and whether each Internet of Things device is started in each target time period.

[0045] According to one embodiment of the present invention, each historical period can be one day, one week, etc., and the present invention does not limit the duration of the historical period. Multiple target time periods when the target person is in the target area can be determined, for example, multiple target time periods when the target person is in the room in the past multiple days.

[0046] According to one embodiment of the present invention, within each target time period, the historical startup time of each IoT device can be determined, so that the startup order of the IoT devices within each target time period can be determined based on the chronological order of the historical startup times. For example, it can be determined which IoT devices are started each time the target person enters the room, as well as the startup order of these IoT devices.

[0047] According to an embodiment of the present invention, the demand coefficient of each IoT device is determined according to the startup sequence of each IoT device in each target time period and whether each IoT device is started in each target time period, including: determining the demand coefficient D of the i-th IoT device according to formula (1): i ,

[0048]

[0049] Where N is the number of target time periods, N start,i is the number of startups of the ith IoT device in multiple target time periods, M is the number of IoT devices in the target area, O i,j is the startup sequence of the i-th IoT device in the j-th target time period, i≤M, j≤N, and i, M, j and N are all positive integers.

[0050] According to one embodiment of the present invention, in formula (1), It is the number of times the i-th IoT device is started and the number of target time periods, which can be taken as the ratio of the number of times the target person starts the i-th IoT device after entering the target area to the total number of times the target person enters the target area. The higher the ratio, the higher the probability that the target person starts the i-th IoT device after entering the target area, which also indicates that the target person has the highest demand for the i-th IoT device.

[0051] According to one embodiment of the present invention, in formula (1), It can be used to indicate the urgency of the target person to start the i-th IoT device after entering the target area for the jth time, that is, the earlier the start sequence (O i,j The smaller the value of ), the larger the ratio is. Therefore, this ratio can describe the target person’s urgency to start the i-th IoT device through the startup sequence of the i-th IoT device, and can also represent the target person’s demand for the i-th IoT device. It can represent the average value of the target person's urgency to start the i-th IoT device after entering the target area multiple times. It can be used to describe the target person's overall urgency to start the i-th IoT device, and can also describe the target person's demand for the i-th IoT device.

[0052] According to one embodiment of the present invention, the probability of the target person activating the i-th IoT device after entering the target area is multiplied by the target person's overall urgency to activate the i-th IoT device to obtain the demand coefficient of the i-th IoT device, which can be used to describe the target person's demand for the i-th IoT device. Based on the above method, the demand coefficient of each IoT device can be determined.

[0053] In this way, the target person's demand for the IoT device can be comprehensively described by the probability of starting the IoT device after entering the target area and the urgency of starting the IoT device, thereby obtaining the demand coefficient of the IoT device and improving the comprehensiveness and accuracy of the demand coefficient.

[0054] According to one embodiment of the present invention, in step S103, the target IoT device for starting the screen and audio components can be determined based on the demand coefficient. For example, each IoT device can be sorted according to the demand coefficient of each IoT device, and the IoT device ranked first can be used as the target IoT device. After the IoT device ranked first is set, the IoT device ranked second can be used as the target IoT device, and so on.

[0055] According to one embodiment of the present invention, in step S104, the screen and audio components of the target IoT device may be started, and based on the type and function of the target IoT device, the audio file played by the audio component may be determined. For example, the audio file may be a prompt audio message prompting the target person to set up the target IoT device.

[0056] According to one embodiment of the present invention, an audio file to be played through an audio component is determined according to the type and function of a target IoT device, including: starting a corresponding environmental parameter detection component according to the type of the target IoT device; obtaining at least one type of environmental parameter through the environmental parameter detection component at multiple moments within a preset time period; determining a startup requirement coefficient of a function of a type corresponding to the environmental parameter according to at least one type of environmental parameter; determining a function to be started according to the startup requirement coefficients of each function; generating a prompt message according to the function to be started, and generating an audio file corresponding to the prompt message; and playing the audio file through the audio component.

[0057] According to one embodiment of the present invention, the corresponding environmental parameter detection component can be started according to the type of the target IoT device. The environmental parameter detection component can be set on the target IoT device, or the environmental parameter detection components corresponding to multiple IoT devices can be set together. The present invention does not limit this. In the example, if the type of the target IoT device is an air conditioner, the environmental parameter detection component may include an air temperature detection component and an air humidity detection component. If the type of the target IoT device is a speaker, the environmental parameter detection component is a noise detection component. The present invention does not limit the specific type of the environmental parameter detection component.

[0058] According to an embodiment of the present invention, the preset time period may be 10 seconds, 30 seconds, 1 minute, etc., and the present invention does not limit the length of the preset time period. The time interval between adjacent moments is 1 second, 0.1 second, etc., and the present invention does not limit the time interval between adjacent moments. Environmental parameters can be obtained through environmental parameter detection components, such as the above example, air temperature and air humidity at multiple moments can be obtained, or noise intensity at multiple moments can be obtained.

[0059] According to one embodiment of the present invention, the startup demand coefficient can be used to describe whether the environment in the target area is suitable. If the environment is not suitable, for example, the air humidity is high or the air temperature is high, there is a high demand for starting the dehumidification function of the air conditioner for dehumidification, or starting the temperature control function of the air conditioner for cooling.

[0060] According to one embodiment of the present invention, determining the startup requirement coefficient of the function of the type corresponding to the environmental parameter according to at least one type of environmental parameter includes: determining the startup requirement coefficient D of the kth type of function corresponding to the environmental parameter according to formula (2): start,k ,

[0061]

[0062] Among them, p k,s,t is the measured value of the sth environmental parameter corresponding to the kth type of function at the tth moment, p p,k,s is the preset value of the sth environmental parameter corresponding to the kth type of function, p k,s,t+1 is the measured value of the sth environmental parameter corresponding to the kth type of function at the t+1th moment, max is the maximum value function, m is the number of types of environmental parameters corresponding to the kth type of function, n is the number of moments in the preset time period, s, t, m and n are all positive integers.

[0063] According to one embodiment of the present invention, in formula (2), is the relative deviation between the sth environmental parameter corresponding to the kth type of function (for example, the air temperature corresponding to the temperature control function) and the preset value, The average relative deviation of multiple environmental parameters corresponding to the kth type of function at multiple times. The larger the average relative deviation, the more the environmental parameters corresponding to the kth type of function deviate from the appropriate situation, and the higher the demand for starting the kth function.

[0064] According to one embodiment of the present invention, in formula (2), Indicates the relative change of environmental parameters at adjacent moments, It represents the average relative change of multiple environmental parameters at multiple moments. The larger the average relative change, the worse the overall stability of the multiple environmental parameters. For example, if the noise increases rapidly or the noise fluctuates, the higher the demand for starting the kth function.

[0065] According to one embodiment of the present invention, the maximum value of the above average relative deviation and the average relative change may be taken as the activation demand coefficient of the k-th type of function to describe the overall demand for the k-th type of function.

[0066] In this way, the overall demand of the current environment for starting various functions can be described by the average relative deviation of the environmental parameters corresponding to various functions and the maximum value of the average relative change, thereby improving the objectivity and accuracy of the startup demand coefficient.

[0067] According to one embodiment of the present invention, the function with the highest startup demand coefficient can be determined as the function to be started. After the function with the highest startup demand coefficient is set, the function with the second highest startup demand coefficient can be determined as the function to be started, and so on. Of course, multiple functions can also be set at one time, for example, one or more functions with a startup demand coefficient greater than or equal to a preset coefficient threshold are all determined as functions to be started, for example, the temperature adjustment function and the dehumidification function of the air conditioner are both set as functions to be started.

[0068] According to one embodiment of the present invention, a prompt message can be generated according to the function to be started, and an audio file corresponding to the prompt message can be generated. For example, when the air temperature is high, a prompt message can be generated in combination with the detected air temperature and the name of the function to be started, such as a text-type prompt message such as "A high temperature is detected, and it is recommended to start the temperature control function to cool down". The prompt message can be generated by a text generation model, and an audio file can be generated using the prompt message, that is, an audio file that reads out the pronunciation of each word in the prompt message, and the audio file can be played through an audio component.

[0069] According to one embodiment of the present invention, in step S105, the function menu displayed on the screen is determined based on the audio file and the type and function of the target IoT device, that is, the prompt message corresponding to the audio file may involve one or more functions, and the function menu to be displayed can be selected from the function menus of multiple functions of the target IoT device based on these functions. For example, the function menu for setting the temperature of the air conditioner, the function menu for setting the dehumidification function, and the function menu for setting the ventilation function can be selected to display the function menu for setting the temperature and the function menu for setting the dehumidification function.

[0070] According to one embodiment of the present invention, in step S106, based on the prompt message corresponding to the audio file, the function type of the recommended setting, that is, the name of the target function to be started, can be determined. The setting method can also be determined based on the selection button in the function menu, for example, inputting a temperature number, or a button for heating up or cooling down, etc. The recommended value of the environmental parameter can also be determined, for example, the preset value of the above environmental parameter. That is, the recommended setting information including information such as the target function, environmental parameters, setting method, and recommended value is obtained. For example, it is recommended to start the temperature adjustment function of the air conditioner, and the recommended setting information of adjusting the current air temperature value of 30°C to the recommended value of 25°C by inputting temperature data.

[0071] According to an embodiment of the present invention, in step S107, the recommended text information is text that can be displayed on the screen and is used to prompt the target person to make settings. The recommended text information is displayed in a specific manner to obtain subtitles.

[0072] According to one embodiment of the present invention, recommended text information is generated based on the recommended setting information and the audio file, including: determining a target function to be started of a target IoT device, environmental parameters corresponding to the target function, and recommended values ​​of the environmental parameters of the recommended settings based on the recommended setting information; processing the target function, the environmental parameters corresponding to the target function, and the recommended values ​​of the environmental parameters through a trained text generation model to obtain a pending recommended text; obtaining an audio text of the audio file; verifying the pending recommended text through the audio text; if the verification passes, determining the pending recommended text as the recommended text information; otherwise, retraining the text generation model.

[0073] According to one embodiment of the present invention, the target function to be activated, the environmental parameters corresponding to the target function, and the recommended values ​​of the recommended environmental parameters can be parsed from the above-mentioned recommended setting information, and can be processed by the trained text generation model to obtain the pending recommended text, that is, the text that integrates and polishes the name of the target function, the current environmental parameters, and the recommended values ​​of the environmental parameters. For example, "It is detected that the current temperature is high, reaching 30°C, and the comfort level is not good. It is recommended that you use the temperature adjustment function to set the temperature to 25°C to lower the temperature and improve the comfort level."

[0074] According to one embodiment of the present invention, the text generation model may be a deep learning neural network model, for example, a recursive neural network model, and the present invention does not limit the specific type of the text generation model. Before using the text generation model for the above processing, training may be performed.

[0075] According to one embodiment of the present invention, the training step of the text generation model includes: obtaining a sample function, a sample environment parameter corresponding to the sample function, and a sample recommendation value; processing the sample function, the sample environment parameter and the sample recommendation value through the text generation model to obtain a sample recommendation text; determining a reference adjustment direction description text according to the sample environment parameter and the sample recommendation value; determining a reference environment description text according to the sample environment parameter; determining a reference environment adjustment description text according to the sample recommendation value; determining a sample adjustment direction description text, a sample environment description text and a sample environment adjustment description text according to the sample recommendation text; obtaining a verification text according to the sample function, the sample environment parameter corresponding to the sample function, and the sample recommendation value; determining a loss function of the text generation model according to the verification text, the sample recommendation text, the reference adjustment direction description text, the reference environment description text, the reference environment adjustment description text, the sample adjustment direction description text, the sample environment description text and the sample environment adjustment description text; training the text generation model according to the loss function of the text generation model to obtain a trained text generation model.

[0076] According to one embodiment of the present invention, the sample function can be any function of any IoT device, the sample environmental parameters can be measured environmental parameters or manually set environmental parameters, and the sample recommended values ​​can be recommended values ​​of the manually set environmental parameters. The above information can be processed through a text generation model to obtain a sample recommendation text.

[0077] According to one embodiment of the present invention, the above information and sample recommendation text can be analyzed to determine the accuracy of the sample recommendation text, thereby determining the accuracy of the text generation model. For example, the reference adjustment direction description text can be determined based on the sample environment parameter and the sample recommendation value. For example, if the sample environment parameter is 30°C and the sample recommendation value is 25°C, the reference adjustment direction description text is a text such as "lowering" or "cooling down". In addition, the reference environment description text can be determined based on the sample environment parameter. For example, if the sample environment parameter is 30°C, the reference environment description text is a text describing a higher ambient temperature, such as "high temperature", "poor comfort", "high temperature" and other texts. Further, the reference environment adjustment description text can also be determined according to the sample recommendation value. For example, if the sample recommendation value is 25°C, the reference environment adjustment description text is a text such as "cooling", "cooling down", "improving comfort" and other texts.

[0078] According to one embodiment of the present invention, sample adjustment direction description text, sample environment description text and sample environment adjustment description text can be determined based on sample recommendation text, that is, text describing the adjustment direction, current environment and adjusted environment can be parsed from the sample recommendation text.

[0079] According to one embodiment of the present invention, a verification text can also be obtained based on the sample function, the sample environment parameters corresponding to the sample function, and the sample recommended value. The verification text does not need to be polished, and only needs to connect these three types of information with a simple sentence, for example, "The current temperature is 30°C, it is recommended that you use the temperature adjustment function to set the temperature to 25°C."

[0080] According to an embodiment of the present invention, based on the above information, the error existing in the sample recommendation text generated by the text generation model can be determined, so as to determine the loss function of the text generation model through the error.

[0081] According to one embodiment of the present invention, the loss function of the text generation model is determined based on the verification text, the sample recommendation text, the reference adjustment direction description text, the reference environment description text, the reference environment adjustment description text, the sample adjustment direction description text, the sample environment description text and the sample environment adjustment description text, including: determining the verification semantic information of the verification text, the sample recommendation semantic information of the sample recommendation text, the reference adjustment direction semantic information of the reference adjustment direction description text, the reference environment semantic information of the reference environment description text, the reference environment adjustment semantic information of the reference environment adjustment description text, the sample adjustment direction semantic information of the sample adjustment direction description text, the sample environment semantic information of the sample environment description text, and the sample environment adjustment semantic information of the sample environment adjustment description text through a semantic recognition model; and determining the loss function LOSS of the text generation model according to formula (3).

[0082]

[0083] Among them, S RD S is the reference adjustment direction semantic information. SD Adjust the direction semantic information for the sample, (S RD ) T For S RD The transposed vector, sim p is the preset similarity threshold, S RE is the reference environment semantic information, (S RE ) T For S RE The transposed vector, S SE is the sample environment semantic information, S RA Adjust semantic information for the reference environment, (S RA ) T For S RA The transposed vector, S SA Adjust the description text for the sample environment, S SR Recommend semantic information for samples, S V To verify the semantic information, w1 and w2 are preset weights, and if is a conditional function.

[0084] According to one embodiment of the present invention, the semantic recognition model can be a recursive neural network model, which can be used to determine the semantic information of various information in vector form, namely, sample recommendation semantic information of sample recommendation text, reference adjustment direction semantic information of reference adjustment direction description text, reference environment semantic information of reference environment description text, reference environment adjustment semantic information of reference environment adjustment description text, sample adjustment direction semantic information of sample adjustment direction description text, sample environment semantic information of sample environment description text, and sample environment adjustment semantic information of sample environment adjustment description text.

[0085] According to one embodiment of the present invention, in formula (3), the conditional function Indicated in In the case of RE -S SE |+w2|S RA -S SA |, otherwise, the conditional function value is Indicates that the cosine similarity between the reference adjustment direction semantic information and the sample adjustment direction semantic information is higher than or equal to a preset similarity threshold, that is, the semantic similarity between the reference adjustment direction description text and the sample adjustment direction description text is high, and the two are synonyms, that is, the sample adjustment direction description text in the sample recommendation text generated by the text generation model is correct. For example, the meaning of the sample adjustment direction description text and the reference adjustment direction description text is to lower the temperature. In this case, the overall semantics of the sample recommendation text is not wrong, and only the text describing the current environmental parameters and the adjusted environmental parameters (for example, the text used for polishing) needs to be optimized, that is, to improve the text generation model's text polishing ability, the error between the reference environment semantic information and the sample environment semantic information, and the error between the reference environment adjustment semantic information and the sample environment adjustment description text can be weighted summed, that is, the semantic errors of the texts describing the current environment parameters and the adjusted environment parameters are weighted summed, and the conditional function value is obtained as the loss function, and the loss function is reduced during the training process, so that the semantic errors of the texts describing the current environment parameters and the adjusted environment parameters are reduced, thereby improving the text generation model's text polishing ability.

[0086] According to an embodiment of the present invention, if the cosine similarity between the reference adjustment direction semantic information and the sample adjustment direction semantic information is lower than a preset similarity threshold, that is, the semantic similarity between the reference adjustment direction description text and the sample adjustment direction description text is low, that is, the sample adjustment direction description text in the sample recommendation text generated by the text generation model is wrong. For example, the reference adjustment direction description text means to lower the temperature, and the sample adjustment direction description text means to increase the temperature. In this case, the overall semantics of the sample recommendation text is wrong. SR -S V|Describe the overall semantic error of the sample recommendation text, and use the cosine similarity between the reference environment semantic information and the sample environment semantic information, the cosine similarity between the reference environment adjustment semantic information and the sample environment adjustment description text, and the cosine similarity between the reference adjustment direction semantic information and the sample adjustment direction semantic information as the denominator to amplify the overall semantic error. Using the amplified semantic error as the loss function can improve the training intensity, and the loss function can be reduced during the training process to reduce the semantic error between the sample recommendation semantic information and the verification semantic information, thereby improving the semantic accuracy of the sample recommendation text. It can also improve the cosine similarity between the reference environment semantic information and the sample environment semantic information, the cosine similarity between the reference environment adjustment semantic information and the sample environment adjustment description text, and the cosine similarity between the reference adjustment direction semantic information and the sample adjustment direction semantic information, thereby improving the description accuracy of the text generation model for the adjustment direction and the description accuracy of the current environment parameters and the adjusted environment parameters.

[0087] According to one embodiment of the present invention, the above-mentioned loss function can be back-propagated, and the parameters of the text generation model can be adjusted by the gradient descent method. The above adjustment steps can be performed multiple times, thereby performing multiple trainings to obtain a trained text generation model.

[0088] In this way, the conditional function can be used to determine whether the overall semantics of the sample recommendation text is erroneous, and the different aspects of the text generation model can be improved when there are no errors and when there are errors, that is, the text generation model's ability to polish the text and the accuracy of its description of the text can be improved, thereby improving the accuracy of the text generation model and improving training efficiency.

[0089] According to one embodiment of the present invention, a pending recommendation text can be generated using a trained text generation model. Further, the pending recommendation text can be verified, that is, the audio text of the audio text can be used for verification. The audio text is a text composed of multiple words in an audio file, and its meaning is theoretically similar to the pending recommendation text, but the description of the specific data is not accurate, that is, the audio text is the text corresponding to the audio text that prompts the target person to make settings, which only has a prompting effect, for example, prompting the target person that the current temperature is high and the temperature adjustment function can be turned on to cool down, but it does not involve more details, for example, it does not involve the current specific environmental parameters, nor the recommended values ​​of the environmental parameters.

[0090] According to one embodiment of the present invention, the pending recommended text is verified through audio text, including: determining the target function description text, the current environment description text and the environment adjustment description text according to the pending recommended text information; forming the target function description text, the current environment description text and the environment adjustment description text into a control text; determining the control semantic information of the control text through a semantic recognition model; determining the audio semantic information of the audio text through the semantic recognition model; determining the semantic similarity between the control semantic information and the audio semantic information; and determining that the verification is passed when the semantic similarity is greater than or equal to a preset semantic similarity threshold.

[0091] For example, the target function description text, the current environment description text and the environment adjustment description text can be combined into a control text, for example, "The current temperature is high and the comfort level is poor. We recommend that you use the temperature adjustment function to lower the temperature and improve the comfort level", and obtain its control semantic information. Furthermore, the audio semantic information of the audio text "The temperature is detected to be high. It is recommended to start the temperature adjustment function to cool down" can also be determined. The control semantic information and the audio semantic information are theoretically highly similar, for example, greater than or equal to a preset semantic similarity threshold. If the semantic similarity is greater than or equal to the preset semantic similarity threshold, it is determined that the verification is passed, and the pending recommended text is determined as the recommended text information. Conversely, the text generation model can be retrained to further improve the performance of the text generation model using richer training data, thereby generating more accurate pending recommended text.

[0092] According to one embodiment of the present invention, in step S108, the subtitles displayed in the function menu can be determined based on the recommended text information. For example, the font, font size, color, etc. of the recommended text information can be set to obtain the subtitles and display them in the function menu on the screen.

[0093] According to the subtitle matching and display method based on audio file processing of an embodiment of the present invention, the target person's demand coefficient for multiple IoT devices can be determined by the target person's historical startup time for IoT devices, thereby determining the IoT device to be started, and when the IoT device plays an audio file to prompt the target person to operate, subtitles can be generated, thereby providing operation prompts to the target person, so that the target person can correctly operate the IoT device and improve the convenience of use. When determining the demand coefficient, the probability of the target person starting the IoT device after entering the target area and the urgency of starting the IoT device can be used to comprehensively describe the target person's demand for the IoT device, thereby obtaining the demand coefficient of the IoT device and improving the comprehensiveness and accuracy of the demand coefficient. When determining the startup demand coefficient, the average relative deviation of the environmental parameters corresponding to various functions and the maximum value of the average relative change can be used to describe the overall demand of the current environment for starting various functions, thereby improving the objectivity and accuracy of the startup demand coefficient. When training a text generation model, a conditional function can be used to determine whether the overall semantics of the sample recommendation text is erroneous, and to improve different aspects of the text generation model's capabilities when no errors occur and when errors occur, respectively. That is, the text generation model's ability to polish the text and the accuracy of its description of the text can be improved, thereby improving the accuracy of the text generation model and improving training efficiency.

[0094] Figure 2 A block diagram of a subtitle matching and display system based on audio file processing according to an embodiment of the present invention is exemplarily shown, wherein the system comprises:

[0095] A target area module, used to determine the target area where the target person is located in multiple areas by using an infrared sensor set on the IoT device, wherein each area includes at least one IoT device, and the IoT device includes a screen and an audio component;

[0096] A demand coefficient module is used to determine the demand coefficients of the target personnel for multiple IoT devices based on the historical startup times of the IoT devices in the target area;

[0097] A target IoT device module, used for determining a target IoT device for starting a screen and audio component according to the demand coefficient;

[0098] An audio file module, used to determine the audio file to be played through the audio component based on the type and function of the target IoT device;

[0099] A function menu module, used to determine a function menu displayed on the screen according to the audio file and the type and function of the target IoT device;

[0100] A recommended setting information module, used to generate recommended setting information according to the audio file, the type and function of the target IoT device, and the selection button in the function menu;

[0101] A recommended text information module, used to generate recommended text information according to the recommended setting information and the audio file;

[0102] The subtitle module is used to determine the subtitles displayed in the function menu according to the recommended text information.

[0103] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A subtitle matching and display method based on audio file processing, characterized in that: include: Determine a target area where a target person is located in a plurality of areas by using an infrared sensor disposed on an IoT device, wherein each area includes at least one IoT device, and the IoT device includes a screen and an audio component; Determine the target person’s demand coefficient for multiple IoT devices based on the historical startup time of IoT devices in the target area; Determining target IoT devices for launching screen and audio components based on the demand coefficients; Determine the audio files to be played through the audio component based on the type and capabilities of the target IoT device; Determine a function menu displayed on the screen according to the audio file and the type and function of the target IoT device; Generate recommended setting information according to the audio file, the type and function of the target IoT device, and the selection button in the function menu; generating recommended text information according to the recommended setting information and the audio file; According to the recommended text information, subtitles displayed in the function menu are determined.

2. The subtitle matching and display method based on audio file processing according to claim 1, characterized in that: According to the historical startup time of IoT devices in the target area, the target personnel's demand coefficient for multiple IoT devices is determined, including: Determine, within a plurality of historical periods, a plurality of target time periods during which the target person is within the target area; Determine the historical startup time of each IoT device within each target time period; According to the historical startup time of each IoT device in each target time period, determine the startup order of each IoT device in each target time period; Determine whether each IoT device is started during each target time period; The demand coefficient of each IoT device is determined according to the startup sequence of each IoT device in each target time period and whether each IoT device is started in each target time period.

3. The subtitle matching and display method based on audio file processing according to claim 2 is characterized in that: According to the startup sequence of each IoT device in each target time period, and whether each IoT device is started in each target time period, the demand coefficient of each IoT device is determined, including: According to the formula Determine the demand coefficient D of the ith IoT device i , where N is the number of target time periods, N start,i is the number of startups of the ith IoT device in multiple target time periods, M is the number of IoT devices in the target area, O i,j is the startup sequence of the i-th IoT device in the j-th target time period, i≤M, j≤N, and i, M, j and N are all positive integers.

4. The subtitle matching and display method based on audio file processing according to claim 1, characterized in that: Depending on the type and capabilities of the target IoT device, determine the audio files that are played through the audio component, including: According to the type of target IoT device, start the corresponding environmental parameter detection component; At multiple moments in a preset time period, obtaining at least one type of environmental parameter through the environmental parameter detection component; According to at least one type of environmental parameter, determining a startup demand coefficient of a function of a type corresponding to the environmental parameter; Determine the functions to be started according to the startup demand coefficient of each function; Generate a prompt message according to the function to be started, and generate an audio file corresponding to the prompt message; The audio file is played through the audio component.

5. The subtitle matching and display method based on audio file processing according to claim 1, characterized in that: Determining, according to at least one type of environmental parameter, a startup requirement coefficient of a function of a type corresponding to the environmental parameter, comprising: According to the formula Determine the startup demand coefficient D of the kth type of function corresponding to the environmental parameters start,k , where p k,s,t is the measured value of the sth environmental parameter corresponding to the kth type of function at the tth moment, p p,k,s is the preset value of the sth environmental parameter corresponding to the kth type of function, p k,s,t+1 is the measured value of the sth environmental parameter corresponding to the kth type of function at the t+1th moment, max is the maximum value function, m is the number of types of environmental parameters corresponding to the kth type of function, n is the number of moments in the preset time period, s, t, m and n are all positive integers.

6. The subtitle matching and display method based on audio file processing according to claim 4 is characterized in that: Generating recommended text information according to the recommended setting information and the audio file, including: Determine, according to the recommended setting information, a target function to be activated of the target IoT device, an environmental parameter corresponding to the target function, and a recommended value of the environmental parameter of the recommended setting; The target function, the environmental parameters corresponding to the target function, and the recommended values ​​of the environmental parameters are processed by the trained text generation model to obtain a pending recommended text; Get the audio text of the audio file; Verifying the pending recommendation text through the audio text; If the verification is successful, the pending recommended text is determined as the recommended text information; Otherwise, retrain the text generation model.

7. The subtitle matching and display method based on audio file processing according to claim 6 is characterized in that: The training steps of the text generation model include: Obtain sample functions, sample environment parameters corresponding to the sample functions, and sample recommended values; Processing the sample function, the sample environment parameter and the sample recommendation value through a text generation model to obtain a sample recommendation text; Determining a reference adjustment direction description text according to the sample environmental parameters and the sample recommended values; Determining a reference environment description text according to the sample environment parameters; Determining a reference environment adjustment description text according to the sample recommended value; Determine, according to the sample recommendation text, a sample adjustment direction description text, a sample environment description text, and a sample environment adjustment description text; Obtaining a verification text according to the sample function, the sample environment parameter corresponding to the sample function, and the sample recommended value; Determine a loss function of a text generation model according to the verification text, the sample recommendation text, the reference adjustment direction description text, the reference environment description text, the reference environment adjustment description text, the sample adjustment direction description text, the sample environment description text, and the sample environment adjustment description text; According to the loss function of the text generation model, the text generation model is trained to obtain a trained text generation model.

8. The subtitle matching and display method based on audio file processing according to claim 7 is characterized in that: Determining a loss function of a text generation model according to the verification text, the sample recommendation text, the reference adjustment direction description text, the reference environment description text, the reference environment adjustment description text, the sample adjustment direction description text, the sample environment description text, and the sample environment adjustment description text includes: Determine, by means of a semantic recognition model, verification semantic information of the verification text, sample recommendation semantic information of the sample recommendation text, reference adjustment direction semantic information of the reference adjustment direction description text, reference environment semantic information of the reference environment description text, reference environment adjustment semantic information of the reference environment adjustment description text, sample adjustment direction semantic information of the sample adjustment direction description text, sample environment semantic information of the sample environment description text, and sample environment adjustment semantic information of the sample environment adjustment description text; According to the formula Determine the loss function LOSS of the text generation model, where S RD S is the reference adjustment direction semantic information. SD Adjust the direction semantic information for the sample, (S RD ) T For S RD The transposed vector, sim p is the preset similarity threshold, S RE is the reference environment semantic information, (S RE ) T For S RE The transposed vector, S SE is the sample environment semantic information, S RA Adjust semantic information for the reference environment, (S RA ) T For S RA The transposed vector, S SA Adjust the description text for the sample environment, S SR Recommend semantic information for samples, S V To verify the semantic information, w1 and w2 are preset weights, and if is a conditional function.

9. The subtitle matching and display method based on audio file processing according to claim 6, characterized in that: The pending recommendation text is verified through the audio text, including: According to the pending recommended text information, determine the target function description text, the current environment description text and the environment adjustment description text; The target function description text, the current environment description text and the environment adjustment description text are combined into a comparison text; Determine the comparison semantic information of the comparison text through a semantic recognition model; Determine the audio semantic information of the audio text through the semantic recognition model; determining semantic similarity between the control semantic information and the audio semantic information; When the semantic similarity is greater than or equal to a preset semantic similarity threshold, it is determined that the verification is passed.

10. A subtitle matching and display system based on audio file processing, characterized in that: include: A target area module, used to determine the target area where the target person is located in multiple areas by using an infrared sensor set on the IoT device, wherein each area includes at least one IoT device, and the IoT device includes a screen and an audio component; A demand coefficient module is used to determine the demand coefficients of the target personnel for multiple IoT devices based on the historical startup times of the IoT devices in the target area; A target IoT device module, used for determining a target IoT device for starting a screen and audio component according to the demand coefficient; An audio file module, used to determine the audio file to be played through the audio component based on the type and function of the target IoT device; A function menu module, used to determine a function menu displayed on the screen according to the audio file and the type and function of the target IoT device; A recommended setting information module, used to generate recommended setting information according to the audio file, the type and function of the target IoT device and the selection button in the function menu; A recommended text information module, used to generate recommended text information according to the recommended setting information and the audio file; The subtitle module is used to determine the subtitles displayed in the function menu according to the recommended text information.

Images