Data processing method, decoding end, coding end, system, storage medium and program product

Abstract

The invention relates to a data processing method, a decoding end, a coding end, a system, a storage medium and a program product. The method can be executed by a decoding end, and comprises the following steps: receiving compressed audio data sent by a coding end; obtaining audio features according to compressed audio data, wherein the compressed audio data is obtained by compressing original audio data of the target object by the coding end; and determining an analysis result corresponding to the target object based on the audio feature, the text feature and the classification model, wherein the analysis result is used for indicating classification information of the health state of the target object. Health state analysis is carried out based on the audio features and the text features, semantic information of multiple modes can be more fully mined, and more comprehensive and effective feature support is provided for analysis.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a data processing method, a decoding end, a coding end, a system, a storage medium and a program product. BACKGROUND

[0002] With the rapid development of remote medical technology, medical data processing systems based on audio analysis have become a research hotspot. In the remote medical scene, audio signals usually need to be compressed before transmission, and the compression and decompression process of audio inevitably introduces information loss, affecting the accuracy of subsequent health status diagnosis. Secondly, in the data processing process, single modal data has significant limitations, and audio signals are difficult to reflect the semantic description of patients, while text lacks objective physiological indicator support. SUMMARY

[0003] The present disclosure provides a data processing method, a decoding end, a coding end, a system, a storage medium and a program product.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, executed by a decoding end, and the method comprises:

[0005] receiving compressed audio data sent by a coding end;

[0006] determining an audio feature according to the compressed audio data, the compressed audio data being obtained by compressing original audio data of a target object by the coding end;

[0007] determining an analysis result corresponding to the target object based on the audio feature, a text feature and a classification model, the analysis result being used to indicate classification information of a health status of the target object.

[0008] According to a second aspect of the present disclosure, a data processing method is provided, executed by a coding end, and the method comprises:

[0009] obtaining original audio data and a metadata file of a target object;

[0010] compressing the original audio data to obtain compressed audio data;

[0011] sending the compressed audio data to a decoding end, the compressed audio data being used by the decoding end to determine an audio feature;

[0012] sending first information to the decoding end, the first information being used by the decoding end to determine a text feature;

[0013] receive the analysis result of the target object sent by the decoding end, the analysis result being used to indicate classification information of a health state of the target object, and the analysis result being determined based on the audio feature, the text feature, and a classification model.

[0014] According to a third aspect of the embodiments of the present disclosure, a decoding end is provided, which is configured to perform the data processing method of the first aspect.

[0015] According to a fourth aspect of the embodiments of the present disclosure, an encoding end is provided, which is configured to perform the data processing method of the first aspect.

[0016] According to a fifth aspect of the embodiments of the present disclosure, a data processing system is provided, which includes a decoding end and an encoding end, the decoding end being configured to implement the data processing method of the first aspect, and the encoding end being configured to implement the data processing method of the second aspect.

[0017] According to a sixth aspect of the embodiments of the present disclosure, a storage medium is provided, which stores instructions, when the instructions are executed on a device, causing the device to perform the data processing method of the first aspect or the second aspect.

[0018] According to a seventh aspect of the embodiments of the present disclosure, a computer program product is provided, which includes a computer program and / or instructions, when the computer program and / or the instructions are executed on a device, implementing the data processing method of the first aspect or the second aspect.

[0019] In the above embodiments, the encoding end can send compressed audio data to the decoding end, and the decoding end can perform health state analysis based on the audio feature and the text feature after determining the audio feature and the text feature, so as to more fully mine semantic information of multiple modalities and provide more comprehensive and effective feature support for analysis. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following describes the drawings required for the embodiments, and the following drawings are only some embodiments of the present disclosure, and do not specifically limit the protection scope of the present disclosure.

[0021] Figure 1 is an exemplary schematic diagram of an architecture of a communication system provided according to the embodiments of the present disclosure.

[0022] Figure 2A is an exemplary interactive schematic diagram of a data processing method provided according to the embodiments of the present disclosure.

[0023] Figure 2B is an exemplary interactive schematic diagram of a data processing method provided according to the embodiments of the present disclosure.

[0024] Figure 2C is an example interaction schematic diagram of a data processing method according to an embodiment of the present disclosure.

[0025] Figure 3A is an example flow schematic diagram of a data processing method according to an embodiment of the present disclosure.

[0026] Figure 3B is an example flow schematic diagram of a data processing method according to an embodiment of the present disclosure.

[0027] Figure 3C is an example flow schematic diagram of a data processing method according to an embodiment of the present disclosure.

[0028] Figure 3D is an example flow schematic diagram of a data processing method according to an embodiment of the present disclosure.

[0029] Figure 3E is an example flow schematic diagram of a data processing method according to an embodiment of the present disclosure.

[0030] Figure 4A is an example schematic diagram of a data processing system according to an embodiment of the present disclosure.

[0031] Figure 4B is an example schematic diagram of a feature fusion module according to an embodiment of the present disclosure.

[0032] Figure 5A is an example structure schematic diagram of a decoding end according to an embodiment of the present disclosure.

[0033] Figure 5B is an example structure schematic diagram of an encoding end according to an embodiment of the present disclosure.

[0034] Figure 6A is an example structure schematic diagram of a data processing device according to an embodiment of the present disclosure.

[0035] Figure 6B is an example structure schematic diagram of a chip according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The present disclosure provides a data processing method, a decoding end, an encoding end, a system, a storage medium and a program product.

[0037] In a first aspect, the present disclosure provides a data processing method, executed by a decoding end, comprising:

[0038] receiving compressed audio data sent by an encoding end;

[0039] determine an audio feature according to the compressed audio data, the compressed audio data being obtained by compressing original audio data of the target object at the encoding end;

[0040] determine an analysis result corresponding to the target object based on the audio feature, the text feature, and a classification model, the analysis result being used to indicate classification information of a health state of the target object.

[0041] In the above embodiments, the encoding end can send compressed audio data to the decoding end. The decoding end can determine an audio feature and a text feature, and perform health state analysis based on the audio feature and the text feature. This can more fully mine semantic information of multiple modalities, and provide more comprehensive and effective feature support for analysis.

[0042] In some embodiments of the first aspect, determining the analysis result corresponding to the target object based on the audio feature, the text feature, and the classification model includes:

[0043] obtaining a fusion feature according to the audio feature and the text feature;

[0044] processing the fusion feature based on a pre-trained classification model to determine the analysis result corresponding to the target object.

[0045] In the above embodiments, the audio feature and the text feature are fused, and a classification model is used for processing, thereby realizing comprehensive analysis of multi-modal data. This fusion method can fully utilize the data advantages of the two modalities of audio and text, and improve the accuracy and reliability of health state classification.

[0046] In some embodiments of the first aspect, the method further includes:

[0047] receiving first information sent by the encoding end;

[0048] determining the text feature based on the first information;

[0049] The first information includes any of the following:

[0050] the text feature;

[0051] a metadata file;

[0052] first description information corresponding to the metadata file, the first description information being used to describe content in the metadata file based on a natural language.

[0053] In the above embodiments, the decoding end can directly receive the text features, or can determine the text features by receiving a metadata file or corresponding description information thereof, so that the system can flexibly select a text feature acquisition manner according to an actual application scenario and data transmission condition, and the adaptability and flexibility of the system are enhanced.

[0054] In some embodiments of the first aspect, the first information includes the metadata file.

[0055] The determining of the text features based on the first information includes:

[0056] generating second description information corresponding to the metadata file according to the metadata file, the second description information being used for describing content in the metadata file based on a natural language;

[0057] determining the text features according to the second description information.

[0058] In the above embodiments, the decoding end can generate second description information based on the metadata file and further determine the text features, so that semantic information in the metadata can be effectively extracted and utilized, richer and more comprehensive text features can be provided for health state analysis, and the utilization efficiency of the system for the metadata is improved.

[0059] In some embodiments of the first aspect, the generating of the second description information corresponding to the metadata file according to the metadata file includes:

[0060] parsing the metadata file to obtain metadata content;

[0061] converting the metadata content into a key-value pair prompt;

[0062] generating the second description information according to the key-value pair prompt.

[0063] In the above embodiments, the decoding end can generate second description information by parsing the metadata file and converting the content thereof into a key-value pair prompt, so that description information with unified format and clear semantics can be generated, the quality and availability of the text data are ensured, and a reliable data basis is provided for subsequent text feature extraction and fusion.

[0064] In some embodiments of the first aspect, the first description information is obtained by field filling of metadata content by the encoding end based on a rule matching dynamic template, and the metadata content is obtained by the encoding end by parsing the metadata file.

[0065] In the above embodiment, the metadata content is filled in the fields by using the dynamic template mode of rule matching, which can quickly and efficiently convert the metadata into the first description information, is suitable for processing of structured metadata, and can quickly generate a text description on a device with limited resources, thereby meeting the real-time requirement.

[0066] In some embodiments of the first aspect, the text feature is determined based on a first weight matrix and a first low-rank matrix, the first low-rank matrix being trained based on low-rank adaptation on a first initial low-rank matrix, and the first low-rank matrix being used to adjust parameters of the first weight matrix.

[0067] In the above embodiment, the initial low-rank matrix is trained by using the low-rank adaptation method to obtain an actually used low-rank matrix, which is used to adjust the parameters of the weight matrix, thereby retaining the advantages of the pre-trained model, optimizing the model for a specific task, reducing the parameter quantity and the calculation amount, and effectively improving the training and inference efficiency of the model.

[0068] In some embodiments of the first aspect, the compressed audio data is obtained by quantizing a first feature representation, and the first feature representation is obtained by encoding the original audio data.

[0069] In the above embodiment, the feature representation is obtained by encoding the original audio data, and the compressed audio data is obtained by quantizing the feature representation, thereby ensuring the compression efficiency while retaining the key features of the audio data, reducing the transmission data amount, improving the system transmission efficiency, and ensuring the effectiveness of the audio features.

[0070] In some embodiments of the first aspect, the determining of the audio feature based on the compressed audio data includes:

[0071] determining restored audio data based on the compressed audio data;

[0072] determining the audio feature based on the restored audio data;

[0073] The restored audio data is obtained by decoding and reconstructing the compressed audio data.

[0074] In the above embodiment, the decoding end can first decode the compressed audio data to obtain the restored audio data, and then extract the audio feature, thereby ensuring the accuracy and integrity of the audio feature and providing a reliable basis for subsequent feature fusion and health state analysis.

[0075] In some embodiments combined with the first aspect, in some embodiments, the audio feature is determined based on a second weight matrix and a second low-rank matrix, the second low-rank matrix is trained based on a low-rank adaptation on a second initial low-rank matrix, and the second low-rank matrix is used to adjust parameters of the second weight matrix.

[0076] In the above embodiments, the initial low-rank matrix can be trained by a low-rank adaptation method to obtain an actually used low-rank matrix, which is used to adjust parameters of the weight matrix, thereby retaining the advantages of the pre-trained model and optimizing for specific tasks, reducing the parameter amount and the calculation amount, and effectively improving the training and inference efficiency of the model.

[0077] In some embodiments combined with the first aspect, in some embodiments, determining the fusion feature based on the audio feature and the text feature comprises:

[0078] Based on a cross-attention mechanism, determining a query matrix based on a third weight matrix corresponding to the audio feature, determining a key matrix based on a fourth weight matrix corresponding to the text feature, and determining a value matrix based on a fifth weight matrix corresponding to the text feature;

[0079] Determining a cross-attention result based on the query matrix, the key matrix, and the value matrix;

[0080] Fusing the cross-attention result with the audio feature and the text feature to obtain the fusion feature.

[0081] In the above embodiments, the cross-attention mechanism is used to deeply fuse the audio feature and the text feature, which can automatically learn the correlation between the two modalities and dynamically adjust the weight of feature fusion, so that the fused feature can more accurately reflect the health status of the target object, effectively improving the accuracy and robustness of health status classification.

[0082] In some embodiments combined with the first aspect, in some embodiments, the classification model comprises a Transformer encoder and a classification head, and determining the analysis result corresponding to the target object based on the fusion feature comprises:

[0083] Determining an encoded feature based on the fusion feature based on the Transformer encoder;

[0084] Determining the analysis result based on the encoded feature based on the classification head.

[0085] In the above embodiment, the classification model composed of the Transformer encoder and the classification head can deeply analyze and process the fusion features, effectively capture the key information in the fusion features for classification, and thus obtain accurate health state analysis results.

[0086] In combination with some embodiments of the first aspect, in some embodiments, at least one of the first low-rank matrix, the second low-rank matrix, the third weight matrix, the fourth weight matrix, the fifth weight matrix, and the classification model is obtained based on a loss function.

[0087] In the above embodiment, the related matrix and the model can be trained based on a predetermined loss function, which can effectively guide the training process of the model, enable the model to learn more accurate feature representation and classification boundary, and improve the performance and generalization ability of the entire system.

[0088] In combination with some embodiments of the first aspect, in some embodiments, the computing resource of the encoding end is higher than the computing resource of the encoding end.

[0089] In the above embodiment, it can be ensured that the computationally intensive task can be executed on a device with more optimal computing resources, thereby improving the processing efficiency and performance of the entire system and ensuring efficient operation of the system.

[0090] In a second aspect, the disclosure embodiments provide a data processing method, executed by an encoding end, comprising:

[0091] Obtaining original audio data and a metadata file of a target object;

[0092] Compressing the original audio data to obtain compressed audio data;

[0093] Sending the compressed audio data to a decoding end, the compressed audio data being used by the decoding end to determine audio features;

[0094] Sending first information to the decoding end, the first information being used by the decoding end to determine text features;

[0095] Receiving an analysis result of the target object sent by the decoding end, the analysis result being used to indicate classification information of a health state of the target object, the analysis result being determined based on the audio features, the text features, and a classification model.

[0096] In combination with some embodiments of the second aspect, in some embodiments, the first information includes any of the following:

[0097] The text features;

[0098] The metadata file;

[0099] The first description information corresponding to the metadata file, the first description information being used for describing content in the metadata file based on a natural language.

[0100] With reference to the second aspect, in some embodiments, the method further includes:

[0101] parsing the metadata file to obtain metadata content;

[0102] performing field filling on the metadata content in a manner of a dynamic template matched based on a rule to obtain the first description information.

[0103] In the above embodiments, the metadata content is filled in the field in the manner of the dynamic template matched based on the rule, which can quickly and efficiently generate the first description information, is suitable for processing of structured metadata, and can quickly generate a text description at a resource-limited encoding end to meet real-time requirements.

[0104] With reference to the second aspect, in some embodiments, the method further includes:

[0105] parsing the metadata file to obtain metadata content;

[0106] converting the metadata content into a key-value pair prompt;

[0107] determining third description information corresponding to the metadata file according to the key-value pair prompt;

[0108] determining the text feature according to the third description information.

[0109] In the above embodiments, the metadata content is converted into the key-value pair prompt, the third description information is generated, and the text feature is determined, which can generate a high-quality and semantically complete text feature, provides a more accurate text feature for health status analysis of a decoding end, and improves utilization efficiency of the system on metadata semantic information.

[0110] With reference to the second aspect, in some embodiments, the compressed audio data is obtained by quantizing a first feature representation, and the first feature representation is obtained by encoding the original audio data.

[0111] With reference to the second aspect, in some embodiments, a computing resource of the decoding end is higher than a computing resource of the encoding end.

[0112] In a third aspect, the embodiments of the present disclosure provide a decoding end configured to perform the data processing method of the first aspect.

[0113] In a fourth aspect, an encoding end is provided, which is configured to implement the data processing method of the second aspect.

[0114] In a fifth aspect, a data processing system is provided, which includes a decoding end and an encoding end. The decoding end is configured to implement the data processing method of the first aspect, and the encoding end is configured to implement the data processing method of the second aspect.

[0115] In a sixth aspect, a storage medium is provided, which stores instructions. When the instructions are executed on a device, the device is caused to implement the data processing method of the first aspect or the second aspect.

[0116] In a seventh aspect, a computer program product is provided, which includes a computer program and / or instructions. When the computer program and / or the instructions are executed on a device, the data processing method of the first aspect or the second aspect is implemented.

[0117] It can be understood that the above data processing method, decoding end, encoding end, system, storage medium, and program product are used to execute the method provided in the embodiments of the present disclosure. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be described here.

[0118] The embodiments of the present disclosure provide a data processing method, a decoding end, an encoding end, a system, a storage medium, and a program product. In some embodiments, the data processing method and the information processing method, the audio text fusion analysis method, and the like can be replaced with each other.

[0119] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing some steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation manners in an embodiment can be combined arbitrarily; in addition, the embodiments can be combined arbitrarily, for example, the steps of different embodiments or part or all of the steps of different embodiments can be combined arbitrarily, an embodiment can be combined with the optional implementation manners of other embodiments. In each embodiment of the present disclosure, the terms and / or descriptions between the embodiments are consistent if there is no special description and no logical conflict, and can be referred to each other, and the technical features in different embodiments can be combined to form a new embodiment according to the logical relationship between them.

[0120] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and not as a limitation on the present disclosure.

[0121] In the embodiments of the present disclosure, an element expressed in singular form, such as "a", "an", "the", "said", "the aforementioned", "the foregoing", "this", and the like, unless otherwise specified, can represent "one and only one", or can represent "one or more", "at least one", and the like. For example, in the case of using an article such as "a", "an", "the" in English, the noun after the article can be understood as a singular expression, or can be understood as a plural expression.

[0122] In the embodiments of the present disclosure, "plurality" refers to two or more.

[0123] In some embodiments, the terms "at least one of A or B", "at least one of A and B", "one or more", "a plurality of", "multiple", and the like can be replaced with each other.

[0124] In some embodiments, the description modes such as "at least one of A or B", "A and / or B", "A in one case and B in another case", "A in response to one case and B in response to another case", and the like can include the following technical solutions according to the case: A in some embodiments (A is executed regardless of whether there is a branch of B); B in some embodiments (B is executed regardless of whether there is a branch of A); A and B are selectively executed in some embodiments (A and B are selectively executed); A and B are executed in some embodiments (A and B are both executed). When there are more branches of A, B, C, and the like, it is similar to the above.

[0125] In some embodiments, the description modes such as "A or B" and the like can include the following technical solutions according to the case: A in some embodiments (A is executed regardless of whether there is a branch of B); B in some embodiments (B is executed regardless of whether there is a branch of A); A and B are selectively executed in some embodiments (A and B are selectively executed). When there are more branches of A, B, C, and the like, it is similar to the above.

[0126] The prefix words of "first", "second" and the like in the embodiments of the present disclosure are merely used to distinguish different description objects, and do not constitute limitation on the position, order, priority, quantity or content of the description objects. The description of the description objects should refer to the description in the context of the claims or embodiments, and should not constitute redundant limitation because of the use of the prefix words. For example, the description objects are "fields", and the ordinal words before "fields" in "first field" and "second field" do not limit the position or order between "fields". "First" and "second" do not limit whether the "fields" modified thereby are in the same message, nor do they limit the order of "first field" and "second field". For another example, the description objects are "levels", and the ordinal words before "levels" in "first level" and "second level" do not limit the priority between "levels". For another example, the quantity of the description objects is not limited by the ordinal words, and can be one or more. For example, "first device", wherein the quantity of "devices" can be one or more. In addition, the objects modified by different prefix words can be the same or different. For example, the description objects are "devices", and "first device" and "second device" can be the same device or different devices, and their types can be the same or different. For another example, the description objects are "information", and "first information" and "second information" can be the same information or different information, and their contents can be the same or different.

[0127] In some embodiments, "including A", "containing A", "for indicating A", "carrying A" can be interpreted as directly carrying A, or indirectly indicating A.

[0128] In some embodiments, the terms of "in response to", "in response to determining", "in the case of", "when", "when", "if", and the like can be replaced with each other. These descriptions all mean that the device will make corresponding processing under certain objective circumstances, and do not necessarily limit the time, and do not require the device to have a judgment action when implemented, nor does it mean that there must be other limitations.

[0129] In some embodiments, the terms of "greater than", "greater than or equal to", "not less than", "more than", "more than or equal to", "not less than", "higher than", "higher than or equal to", "not lower than", "above", and the like can be replaced with each other. The terms of "less than", "less than or equal to", "not greater than", "less than", "less than or equal to", "not more than", "lower than", "lower than or equal to", "not higher than", "below", and the like can be replaced with each other.

[0130] In some embodiments, an apparatus and the like can be interpreted as an entity, and can also be interpreted as virtual, and the name thereof is not limited to the name described in the embodiments. The terms of "apparatus", "equipment", "device", "circuit", "network element", "network function", "network device", "function", "node", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject" and the like can be replaced with each other.

[0131] In some embodiments, the data, information and the like can be acquired in compliance with the laws and regulations of the country where the location is.

[0132] In some embodiments, the data, information and the like can be acquired after obtaining the consent of the user.

[0133] In addition, each element, each row or each column in the table of the embodiments of the disclosure can be implemented as an independent embodiment, and any combination of any element, any row or any column can also be implemented as an independent embodiment.

[0134] Figure 1 is an architecture schematic diagram of a data processing system according to the embodiments of the disclosure. As shown in Figure 1 , the data processing system 100 includes a decoding end 101 and an encoding end 102.

[0135] In some embodiments, the decoding end 101 and the encoding end 102 can be referred to as data processing devices. Optionally, the decoding end 101 and the encoding end 102 both have data processing capabilities.

[0136] In some embodiments, the decoding end 101 and the encoding end 102, for example, include at least one of a mobile phone, a wearable device, an Internet of Things device, a car with communication function, a smart car, a Pad, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, a wireless terminal device in smart home, but are not limited thereto.

[0137] In some embodiments, the data processing capability and computing resource of the decoding end 101 are higher than those of the encoding end 102, or the data processing capability and computing resource of the decoding end 101 are superior to those of the encoding end 102.

[0138] It can be understood that the data processing system described in the embodiments of the present disclosure is for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and does not constitute a limitation on the technical solutions proposed by the embodiments of the present disclosure. It can be known by those skilled in the art that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions proposed by the embodiments of the present disclosure are also applicable to similar technical problems.

[0139] The following embodiments of the present disclosure can be applied to Figure 1 The data processing system 100 or part of the subjects shown, but not limited thereto. Figure 1 The subjects shown are examples, and the data processing system can include Figure 1 all or part of the subjects in the above, but also can include Figure 1 other subjects other than the above, the number and form of each subject is arbitrary, each subject can be real or virtual, the connection relationship between each subject is an example, each subject can not be connected or can be connected, and the connection can be in any way, can be direct connection or indirect connection, can be wired connection or wireless connection.

[0140] With the rapid development of telemedicine technology, medical data processing systems based on audio analysis have become a research hotspot. In the telemedicine scene, audio signals usually need to be compressed before transmission, and the compression and decompression process of audio inevitably introduces information loss, affecting the accuracy of subsequent health status diagnosis. Secondly, in the data processing process, single modal data has significant limitations, and audio signals are difficult to reflect the semantic description of patients, while text lacks objective physiological indicator support. The fusion of multi-modal data can make up for the shortcomings of single modal data, by integrating audio signals and structured metadata, the supplement of metadata can effectively make up for the information loss, and build a more comprehensive diagnostic information system to ensure the robustness of the diagnosis system.

[0141] Existing technologies focus more on the analysis of original audio signals, and there is a lack of research on the application of audio signals compressed and transmitted in health diagnosis. Although multi-modal fusion technology has been initially applied in the field of health diagnosis, in the aspect of metadata processing, the expression and modeling of its semantic layer are often ignored, resulting in limited feature representation ability after fusion. In addition, audio features and text features exist in different metric spaces, and traditional methods are difficult to achieve effective cross-modal alignment, resulting in redundant or semantic loss of fused feature information.

[0142] Therefore, there is an urgent need for an efficient data processing system that can realize cross-modal deep semantic alignment of audio and metadata for compressed and decompressed audio signals, and meet the real-time and high-accuracy requirements in clinical scenarios.

[0143] In some embodiments of the present disclosure, a data processing method based on audio and metadata fusion under an end-to-end processing architecture is provided. By integrating feature extraction and feature fusion technologies, efficient data processing is realized according to health-related audio signals and metadata, and the health status of a user is predicted.

[0144] Figure 2A is an interaction schematic diagram of the data processing method according to an embodiment of the present disclosure. As shown in Figure 2A The present disclosure relates to a data processing method, and the method comprises:

[0145] In step S2101, the encoding end obtains original audio data and a metadata file of a target object.

[0146] In some embodiments, the target object can refer to a user to be analyzed for health status.

[0147] In some embodiments, the encoding end can not be limited to obtaining the original audio data and the metadata file of the target object, but can also obtain the original audio data and the metadata file of one or more other objects.

[0148] In some embodiments, the original audio data includes at least one of the following of the target object: lung sound; heart sound; cough sound. The original audio data can also include, but is not limited to, intestinal sound, joint friction sound, snoring sound, footstep sound, etc., and can be obtained according to actual needs by those skilled in the art.

[0149] In some embodiments, the metadata file includes at least one of the following of the target object: age; age group; gender; height; weight; respiratory system condition; fever condition; muscle pain condition.

[0150] In some embodiments, the metadata file is an Extensible Markup Language (XML) format file. Alternatively, the metadata file can also be a file in other formats, such as a file in JSON, YAML, CSV, etc., and the present disclosure is not limited thereto. The content in the metadata file can be non-natural language content, which is, for example, a computer-readable non-key-value type structured data format file.

[0151] In some embodiments, after the encoding end obtains the original audio data and the metadata file of the target object, it can perform step S2102 and step S2106.

[0152] In some embodiments, steps S2106 to S2110 in the embodiments of the present disclosure can be replaced by steps S2201 to S2205 in the embodiments shown in FIG. 22, or replaced by steps S2301 to S2304 in the embodiments shown in FIG. 23. Figure 2B In some embodiments, steps S2106 to S2110 in the embodiments of the present disclosure can be replaced by steps S2201 to S2205 in the embodiments shown in FIG. 22, or replaced by steps S2301 to S2304 in the embodiments shown in FIG. 23. Figure 2C In some embodiments, steps S2106 to S2110 in the embodiments of the present disclosure can be replaced by steps S2201 to S2205 in the embodiments shown in FIG. 22, or replaced by steps S2301 to S2304 in the embodiments shown in FIG. 23.

[0153] Optionally, the decoding end and the encoding end can determine to perform steps S2106 to S2110, or steps S2201 to S2205, or steps S2301 to S2304 according to respective computing resources and data processing capabilities.

[0154] For example, if the encoding end determines that the computing resources are sufficient, the encoding end can not need to send the metadata file to the decoding end to enable the decoding end to determine the text matrix by itself, and the encoding end can determine the text features based on steps S2201 to S2205 and send the text features to the decoding end.

[0155] In step S2102, the encoding end compresses the original audio data to obtain compressed audio data.

[0156] In some embodiments, the encoding end can use a preset algorithm or a pre-trained AI model to compress the original audio data to obtain the compressed audio data.

[0157] In some embodiments, the encoding end is deployed with a pre-trained audio compression model.

[0158] In some embodiments, the encoding end invokes the audio compression model through a preset interface.

[0159] In some embodiments, the audio compression model is used to compress the audio data, and the data amount of the compressed audio data is less than that of the audio data before compression.

[0160] In some embodiments, the compressed audio data is obtained by quantizing a first feature representation, and the first feature representation is obtained by encoding the original audio data.

[0161] In some embodiments, the encoding end includes a pre-trained audio encoder, and the audio encoder is configured to generate a feature representation of the original audio data and quantize the feature representation to obtain the compressed audio data.

[0162] In some embodiments, the audio encoder can include a full convolutional encoder network, which can generate a latent feature representation of an audio signal, and then input the latent feature representation into a single quantizer for quantization to generate a discretized feature representation, thereby achieving efficient compression of the audio signal.

[0163] In some embodiments, the audio encoder can be, for example, a WavTokenizer-Encoder, or any other encoder capable of implementing the corresponding function.

[0164] At step S2103, the encoding end sends the compressed audio data to the decoding end.

[0165] In some embodiments, the encoding end can send compressed audio data of multiple objects to the decoding end. Optionally, each compressed audio data can have an object identifier uniquely identifying the object of the compressed audio data.

[0166] At step S2104, the decoding end determines the restored audio data according to the compressed audio data.

[0167] In some embodiments, the decoding end can utilize a preset algorithm, or a pre-trained model, to determine the restored audio data according to the compressed audio data.

[0168] In some embodiments, the restored audio data can refer to the data obtained by the decoding end decoding the compressed audio data.

[0169] In some embodiments, a pre-trained audio reconstruction model is deployed in the decoding end. Optionally, the audio reconstruction model can also be referred to as a decoder.

[0170] In some embodiments, the decoding end invokes the audio reconstruction model through a preset interface.

[0171] In some embodiments, if the decoding end receives multiple compressed audio data, the decoding end can determine the compressed audio data corresponding to a target object based on the object identifier of the target object from the multiple compressed audio data, and perform restoration.

[0172] In some embodiments, the audio reconstruction model is used to restore the compressed audio data. Optionally, the audio reconstruction model is trained to restore the compressed audio data as close to the original audio data as possible.

[0173] In some embodiments, the audio reconstruction model can be, for example, a WavTokenizer-Decoder.

[0174] In some embodiments, the audio reconstruction model and the audio compression model can be obtained by joint training. Optionally, the audio reconstruction model and the audio compression model can also be obtained by separate training.

[0175] For example, the audio compression model and the audio reconstruction model can be a jointly trained generative adversarial model, in which the audio compression model is configured to compress the original audio data as small as possible, i.e., to generate smaller audio compression data, and the audio reconstruction model is configured to generate restored audio data as close to the original audio data as possible.

[0176] At step S2105, the decoding end determines the audio feature based on the restored audio data.

[0177] In some embodiments, the decoding end can determine the audio feature based on the restored audio data by using a preset algorithm or a pre-trained AI model.

[0178] In some embodiments, the audio feature is determined based on a second weight matrix and a second low-rank matrix, the second low-rank matrix being trained based on the low-rank adaptation on a second initial low-rank matrix, and the second low-rank matrix being used to adjust the parameters of the second weight matrix.

[0179] Optionally, the parameters of the second weight matrix are fixed during the training process. Optionally, the first initial low-rank matrix can be a matrix without training or with default parameters.

[0180] In some embodiments, a pre-trained audio feature extraction model is deployed in the decoding end.

[0181] In some embodiments, the decoding end calls the audio feature extraction model through a preset interface.

[0182] In some embodiments, the audio feature extraction model is used to extract the audio feature from the restored audio data.

[0183] In some embodiments, the audio feature extraction model can be, for example, a Whisper acoustic model.

[0184] In some embodiments, the audio feature extraction model can adopt an encoder-decoder architecture. Optionally, the encoder converts the audio signal (i.e., the restored audio data) into a mel-spectrogram and then passes it through a multi-layer Transformer encoder to capture long-range dependencies of the audio sequence. Optionally, the decoder generates a text sequence based on the output of the encoder through a cross-attention mechanism. The output encoder hidden state takes the last layer hidden state as the audio feature representation and maps it to a representation space with a dimension of 768 through a linear projection layer.

[0185] For example, the audio feature extraction model extracting the audio feature can be represented as: z a = Projection(f a (x a )); where x a is the restored audio data, fa (·) denotes an audio feature extraction model, z a is an audio feature representation.

[0186] In some embodiments, the audio feature extraction is constituted by a second weight matrix and a second low-rank matrix.

[0187] In some embodiments, the audio feature extraction model comprises a second weight matrix and a second low-rank matrix.

[0188] For example, if the second weight matrix is denoted as W ∈ R h×k , two lightweight trainable low-rank matrices such as A ∈ R r×k , B ∈ R h×r (i.e. the second low-rank matrix) can be introduced to approximate the weight update. During training, the second weight matrix remains frozen, only A and B are updated, reducing the amount of trainable parameters. Wherein, the forward propagation process can be denoted as F = Wx + ΔWx = Wx + BAx, ΔWx is the adjustment amount for the second weight matrix, which is equal to the adjustment amount BAx for the second low-rank matrix, and F can represent the trained audio feature extraction model.

[0189] Step S2106, the encoding end sends the metadata file to the decoding end.

[0190] In some embodiments, the encoding end can send the metadata file of multiple objects to the decoding end. Optionally, each metadata file can have an object identifier uniquely identifying the object of the compressed audio data.

[0191] Step S2107, the decoding end parses the metadata file to obtain the metadata content.

[0192] In some embodiments, if the decoding end receives multiple compressed audio data, the decoding end can determine the compressed audio data corresponding to the target object based on the object identifier of the target object from the multiple compressed audio data, and restore the compressed audio data.

[0193] In some embodiments, the decoding end can be pre-deployed with components for parsing the metadata file, such as Python standard library, extension library lxml, etc. Optionally, the decoding end can call Python xml.etree.ElementTree to parse the metadata file to extract the metadata content.

[0194] For example, taking the metadata file as an XML file, the decoding end can call Python xml.etree.ElementTree to obtain the root element, and traverse the child elements (such as patient information, examination indicators, etc.) in the root element, and then output the tag name and text content of each sub-element in the metadata file, such as "tag: {child.tag}, content: {child.text}".

[0195] In step S2108, the decoding end converts the metadata content into a key-value pair prompt.

[0196] In some embodiments, the decoding end can convert the tag name and text content of each sub-element in the metadata file into a key-value pair prompt.

[0197] In some embodiments, the key-value pair prompt is used to indicate the second description information corresponding to the metadata file generated by the generation model.

[0198] For example, in one metadata instance, the key-value pair prompt is as follows: "Patient information: gender: male, age: 45 years old, respiratory disease: yes. Please summarize the above information in natural language".

[0199] In step S2109, the decoding end determines the second description information corresponding to the metadata file according to the key-value pair prompt.

[0200] In some embodiments, the decoding end can use a preset algorithm or a pre-trained AI model to determine the second description information corresponding to the metadata file according to the key-value pair prompt.

[0201] In some embodiments, the decoding end is deployed with a generation model.

[0202] In some embodiments, the decoding end calls the generation model through a preset interface.

[0203] In some embodiments, the second description information is used to describe the content in the metadata file based on natural language.

[0204] In some embodiments, the generation model is used to generate the second description information corresponding to the metadata file. Optionally, the generation model can be used to convert key-value pair type data into natural language description information.

[0205] In some embodiments, the generation model can be, for example, a BioGPT language model.

[0206] In an example, if the key-value pair prompt is "Patient information: gender: male, age: 45 years old, respiratory disease: yes. Please summarize the above information in natural language", the second description information can be, for example, "The patient is a 45-year-old male with respiratory disease".

[0207] At step S2110, the decoding end determines the text feature according to the second description information.

[0208] In some embodiments, the decoding end can determine the text feature according to the second description information by using a preset algorithm or a pre-trained AI model.

[0209] In some embodiments, the text feature is determined based on the first weight matrix and the first low-rank matrix, the first low-rank matrix being trained based on the low-rank adaptation on the first initial low-rank matrix, and the first low-rank matrix being used to adjust the parameters of the first weight matrix.

[0210] Optionally, the parameters of the first weight matrix are constant during the training process. Optionally, the first initial low-rank matrix can be a matrix without training or with default parameters.

[0211] In some embodiments, the decoding end is deployed with a text feature extraction model.

[0212] In some embodiments, the decoding end calls the text feature extraction model through a preset interface.

[0213] In some embodiments, the text feature extraction model is used to extract the text feature from the second description information.

[0214] In some embodiments, the text feature extraction model can be, for example, a RoBERTa text model.

[0215] In some embodiments, the text feature extraction model can be based on a Transformer encoder, which is composed of multiple encoding layers, each of which is based on a self-attention mechanism to calculate the output representation of each word, and a feedforward neural network further processes the output of the self-attention layer. The hidden state of the last layer of the output encoder is taken as the vector at the first position of the sequence as the text feature representation.

[0216] For example, the text feature extraction model extracting the text feature can be represented as: z t = f t (x t ); where x t represents the second description information, f t (·) represents the text base model, and z t is the text feature representation. Optionally, the dimension of the text feature can be 768.

[0217] In some embodiments, the text feature extraction model can be composed of the first weight matrix and the first low-rank matrix.

[0218] In some embodiments, the text feature extraction model includes the first weight matrix and the first low-rank matrix.

[0219] For example, similar to the audio feature extraction model, if the first weight matrix is denoted as W ∈ R h×k , two light-weight trainable low-rank matrices, such as A ∈ R r×k , B ∈ R h×r (i.e., the first low-rank matrix) can be introduced to approximate the weight update. During training, the first weight matrix remains frozen, and only A and B are updated, reducing the trainable parameter amount. Wherein, the forward propagation process can be denoted as F = Wx + ΔWx = Wx + BAx, ΔWx is the adjustment amount for the first weight matrix, which is equal to the adjustment amount BAx for the first low-rank matrix.

[0220] In step S2111, the decoding end determines a query matrix based on the third weight matrix corresponding to the audio feature, determines a key matrix based on the fourth weight matrix corresponding to the text feature, and determines a value matrix based on the fifth weight matrix corresponding to the text feature based on the cross-attention mechanism.

[0221] In some embodiments, the third weight matrix, the fourth weight matrix, and the fifth weight matrix can be obtained through training.

[0222] In some embodiments, the decoding end can be deployed with a pre-trained feature fusion model, which includes the third weight matrix, the fourth weight matrix, and the fifth weight matrix. Optionally, the feature fusion model is used to fuse the text feature and the audio feature based on the cross-attention mechanism.

[0223] In some embodiments, the decoding end executes step S2111 and the subsequent steps after determining the audio feature and determining the text feature. Optionally, the decoding end inputs the audio feature and the text feature into the feature fusion model to obtain the fused feature after determining the audio feature and determining the text feature.

[0224] In some embodiments, the text feature can be obtained based on steps S2106 to S2110, or can be obtained based on steps S2201 to S2205, or can be obtained based on steps S2301 to S2304.

[0225] In some embodiments, the audio feature can be linearly transformed based on the third weight matrix to obtain the query matrix.

[0226] In some embodiments, the text feature can be linearly transformed based on the fourth weight matrix to obtain the key matrix.

[0227] In some embodiments, the text feature can be linearly transformed based on the fifth weight matrix to obtain the value matrix.

[0228] In some embodiments, determining the query matrix, the key matrix, and the value matrix can be represented as:

[0229] Q = z a W Q ;

[0230] K = z t W K ;

[0231] V = z t W V ;

[0232] wherein Q represents the query matrix, K represents the key matrix, V represents the value matrix, W Q ,W K ,W V is a trainable weight matrix.

[0233] At step S2112, the decoding end determines the cross-attention result according to the query matrix, the key matrix, and the value matrix.

[0234] In some embodiments, the decoding end determines the cross-attention result according to the query matrix, the key matrix, and the value matrix by using a preset algorithm or a pre-trained AI model.

[0235] In some embodiments, the decoding end calculates an attention score and weighted sum according to the query matrix, the key matrix, and the value matrix to obtain the cross-attention result.

[0236] In some embodiments, the decoding end calculates an attention score and weighted sum according to the query matrix, the key matrix, and the value matrix to obtain the cross-attention result can be represented as:

[0237]

[0238] wherein d k is an attention dimension. is used to stabilize the gradient, z att represents the cross-attention result.

[0239] At step S2113, the decoding end fuses the cross-attention result with the audio feature and the text feature to obtain a fused feature.

[0240] In some embodiments, fusing the cross-attention result with the audio feature and the text feature by the decoding end can further enhance information retention.

[0241] In some embodiments, fusing the cross-attention result with the audio feature and the text feature by the decoding end to obtain the fused feature can be represented as: z = concat(z att ,z a ,z t ); wherein za represents an audio feature, z t represents a text feature, z att represents a cross-attention result.

[0242] At step S2114, the decoding end processes the fusion feature based on the pre-trained classification model to determine an analysis result corresponding to the target object.

[0243] In some embodiments, the analysis result is used to indicate classification information of a health state of the target object. Optionally, the analysis result may, for example, include a probability of being healthy, a probability of being unhealthy, a probability of suffering from a respiratory disease, a probability of not suffering from a respiratory disease, and the like.

[0244] In some embodiments, the classification model includes a Transformer encoder and a classification head.

[0245] In some embodiments, the decoding end determines an encoding result from the fusion feature based on the Transformer encoder, and determines the analysis result from the encoding result based on the classification head.

[0246] In some embodiments, the decoding end processes the fusion feature based on the Transformer encoder to obtain an encoding result, and processes the encoding result based on the classification head to obtain the analysis result.

[0247] In some embodiments, the Transformer encoder can include 12 attention heads. Optionally, the Transformer encoder hidden layer dimension is 768.

[0248] In some embodiments, the input feature (i.e., the fusion feature) can be first uniformly mapped to a dimension representation space (such as 768 dimensions) through a linear projection layer, processed through a Transformer encoder (such as 12 attention heads, with a hidden layer dimension of 768), and normalized through a LayerNorm layer, and the vector at the first position in the sequence is taken as the global feature.

[0249] For example, uniformly mapping the fusion feature to a 768-dimensional representation space can be represented as: z' = Projection(z);

[0250] The encoding result obtained by the Transformer encoder can be represented as: y = LayerNorm(Transformer(z'))[0].

[0251] In some embodiments, a classification head composed of three fully connected layers can be used for classification, each fully connected layer introducing a ReLU activation function and a Dropout regularization operation with a probability of 0.5, and finally outputting a probability distribution of the class.

[0252] In some embodiments, based on the classification head, determining the analysis result according to the encoding result can be represented as: p = Classifier(y); where p is a probability distribution of the output category generated by the classification head.

[0253] At step S2115, the decoding end sends the analysis result to the encoding end.

[0254] In some embodiments, the decoding end can send all the analysis results to the encoding end. Alternatively, the decoding end can also send part of the analysis results to the encoding end, for example, the decoding end can send the analysis results including the category with the highest probability or the top N categories with the highest probability to the encoding end, where N can be a predefined integer.

[0255] In some embodiments, after receiving the analysis result, the encoding end can display the analysis result on a user interface, or perform further processing based on the analysis result, which is not limited in the embodiments of the present disclosure.

[0256] In the embodiments of the present disclosure, at least one of the first low-rank matrix (i.e., corresponding to the text feature extraction model), the second low-rank matrix (i.e., corresponding to the audio feature extraction model), the third weight matrix, the fourth weight matrix, the fifth weight matrix (i.e., the feature fusion model), and the classification model can be determined based on a preset loss function.

[0257] In some embodiments, the preset loss function can be represented as:

[0258]

[0259] where N represents the number of samples, C represents the number of categories, y i represents the true label of sample i, p i represents the predicted probability of sample i in category c.

[0260] For example, based on the above loss function, only the classification model can be trained, and the parameters corresponding to the first low-rank matrix, the second low-rank matrix, the third weight matrix, the fourth weight matrix, and the fifth weight matrix can be predefined, or can be trained by using other loss functions or other training methods; or, based on the above loss function, only the third weight matrix, the fourth weight matrix, and the fifth weight matrix can be trained, and the parameters of other modules can be trained by using other loss functions or other training methods, and so on, which is not described here.

[0261] In some embodiments, based on the above loss function, the parameters corresponding to the first low-rank matrix, the second low-rank matrix, the third weight matrix, the fourth weight matrix, the fifth weight matrix, and the classification model can be updated simultaneously to obtain the trained parameters.

[0262] In some embodiments, the parameters of the first low-rank matrix, the second low-rank matrix, the third weight matrix, the fourth weight matrix, the fifth weight matrix, and the parameters corresponding to the classification model can also be updated based on the loss function described above, respectively, to obtain the trained parameters.

[0263] For example, the parameters of the first low-rank matrix, the second low-rank matrix, the third weight matrix, the fourth weight matrix, and the fifth weight matrix can be fixed first, and the classification results (such as including the predicted probability of the training sample in each category) corresponding to one or more training samples can be obtained based on the text feature extraction model, the audio feature extraction model, the feature fusion model, and the classification model. Further, according to the loss function, the parameters of the classification head are updated to obtain the trained classification head, and then the parameters of the first low-rank matrix, the second low-rank matrix, and the like are further updated in turn.

[0264] In some embodiments, the text feature extraction model, the audio feature extraction model, the feature fusion model, and the classification model described above can be trained in the decoding end. Optionally, after the decoding end trains the above-mentioned models, a part of the models can be deployed in the decoding end, and another part of the models (such as model parameters) can be sent to the encoding end for deployment in the encoding end.

[0265] In some embodiments, the text feature extraction model, the audio feature extraction model, the feature fusion model, and the classification model can also be trained in other devices such as cloud servers. After the other devices train the above-mentioned models, these models can be sent to the decoding end for deployment, or a part of them can be sent to the decoding end for deployment, and another part can be sent to the encoding end for deployment.

[0266] In some embodiments, the decoding end is deployed with an audio decoder (such as an audio reconstruction model), an audio feature extraction model, a text feature extraction model, a feature fusion model, and a classification model, and the encoding end is deployed with an audio encoder (such as an audio compression model).

[0267] In some embodiments, the decoding end is deployed with an audio decoder (such as an audio reconstruction model), an audio feature extraction model, a feature fusion model, and a classification model, and the encoding end is deployed with an audio encoder (such as an audio compression model) and a text feature extraction model.

[0268] In some embodiments, reference can be made to the steps and optional implementation manners thereof in other embodiments described before or after the description of the present embodiment, and other related parts of the description, which will not be repeated here.

[0269] Figure 2B is an interaction schematic diagram of the data processing method according to the embodiments of the present disclosure. As shown in Figure 2B ,Figure 2A The steps S2106 to S2110 can be replaced by the following steps:

[0270] In step S2201, the encoding end parses the metadata file to obtain the metadata content.

[0271] In some embodiments, if the encoding end obtains multiple compressed audio data in step S2101, the encoding end can determine the compressed audio data corresponding to the target object based on the object identifier of the target object, and restore the compressed audio data.

[0272] In some embodiments, the encoding end can be pre-deployed with components for parsing the metadata file, such as Python standard library, extension library lxml, etc. Alternatively, the encoding end can call Python xml.etree.ElementTree to parse the metadata file to extract the metadata content.

[0273] For example, taking the metadata file as an XML file, the encoding end can call Python xml.etree.ElementTree to obtain the root element, and traverse the child elements (such as patient information, examination indicators, etc.) in the root element, and then output the tag name and text content of each child element in the metadata file, such as "tag: {child.tag}, content: {child.text}".

[0274] In step S2202, the encoding end converts the metadata content into a key-value pair prompt.

[0275] In some embodiments, the encoding end can convert the tag name and text content of each child element in the metadata file into a key-value pair prompt.

[0276] In some embodiments, the key-value pair prompt is used to instruct the generation model to output the second description information corresponding to the metadata file.

[0277] For example, in one metadata instance, the key-value pair prompt is as follows: "Patient information as follows: gender: male, age: 45 years old, respiratory disease: yes. Please summarize the above information in natural language".

[0278] In step S2203, the encoding end determines the third description information corresponding to the metadata file according to the key-value pair prompt.

[0279] In some embodiments, the encoding end determines the third description information corresponding to the metadata file according to the key-value pair prompt based on a preset algorithm or a pre-trained AI model.

[0280] In some embodiments, the encoding end is deployed with a generation model.

[0281] In some embodiments, the encoding end invokes the generation model through a preset interface.

[0282] In some embodiments, the third description information is used to describe the content in the metadata file based on natural language.

[0283] In some embodiments, the generation module is configured to generate the third description information corresponding to the metadata file.

[0284] In some embodiments, the generation model may, for example, be a BioGPT language model.

[0285] In an example, if the key-value pair prompt is: “Patient information is as follows: gender: male, age: 45 years old, respiratory disease: yes. Please summarize the above information in natural language”, the third description information may, for example, be: “The patient is a 45-year-old male with respiratory disease”.

[0286] At step S2204, the encoding end determines the text feature according to the third description information.

[0287] In some embodiments, the encoding end may, for example, use a preset algorithm or a pre-trained AI model to determine the text feature according to the third description information.

[0288] In some embodiments, the text feature is determined based on the first weight matrix and the first low-rank matrix, the first low-rank matrix being trained based on the low-rank adaptation on the first initial low-rank matrix, and the first low-rank matrix being used to adjust the parameters of the first weight matrix.

[0289] Optionally, the parameters of the first weight matrix remain unchanged during the training process. Optionally, the first initial low-rank matrix may, for example, be a matrix that has not been trained or has default parameters.

[0290] In some embodiments, the encoding end is deployed with a text feature extraction model.

[0291] In some embodiments, the encoding end invokes the text feature extraction model through a preset interface.

[0292] In some embodiments, the text feature extraction model is configured to extract the text feature from the second description information.

[0293] In some embodiments, the text feature extraction model may, for example, be a RoBERTa text model.

[0294] In some embodiments, the text feature extraction model may, for example, be based on a Transformer encoder, which is composed of multiple encoding layers, each of which is based on a self-attention mechanism to calculate the output representation of each word, and a feedforward neural network further processes the output of the self-attention layer. The hidden state of the last layer of the output encoder is output, and the vector at the first position of the sequence is taken as the text feature representation.

[0295] For example, the text feature extraction model extracting the text feature can be represented as: z t = f t (x t ); where f t (·) represents a text base model, and z t is a text feature representation. Optionally, the dimension of the text feature can be 768.

[0296] In some embodiments, the text feature extraction model can be composed of a first weight matrix and a first low-rank matrix.

[0297] In some embodiments, the text feature extraction model includes a first weight matrix and a first low-rank matrix.

[0298] For example, similar to the audio feature extraction model, if the first weight matrix is represented as a weight matrix W ∈ R h×k , two lightweight trainable low-rank matrices such as A ∈ R r×k , B ∈ R h×r (i.e., the first low-rank matrix) can be introduced to approximate the weight update. During training, the first weight matrix remains frozen, and only A and B are updated, reducing the number of trainable parameters. Wherein, the forward propagation process can be represented as: F = Wx + ΔWx = Wx + BAx, ΔWx is the adjustment amount for the first weight matrix, which is equal to the adjustment amount BAx for the first low-rank matrix.

[0299] In step S2205, the encoding end sends the text feature to the decoding end.

[0300] In some embodiments, after receiving the text feature sent by the encoding end, the decoding end can fuse the text feature with the audio feature, and determine the analysis result corresponding to the target object based on the fused feature.

[0301] Figure 2C is an interaction schematic diagram of the data processing method according to the embodiments of the present disclosure. As Figure 2C shown, Figure 2A steps S2106 to S2110 can be replaced by the following steps:

[0302] In step S2301, the encoding end parses the metadata file to obtain the metadata content.

[0303] In some embodiments, if the encoding end obtains multiple compressed audio data in step S2101, the encoding end can determine the compressed audio data corresponding to the target object based on the object identifier of the target object, and restore the compressed audio data.

[0304] In some embodiments, the encoding end can be pre-deployed with components for parsing the metadata file, such as the Python standard library, the extended library lxml, and the like. Alternatively, the encoding end can call the xml.etree.ElementTree of Python to parse the metadata file to extract the metadata content.

[0305] For example, taking the metadata file as an XML file, the encoding end can call the xml.etree.ElementTree of Python to obtain the root element, and traverse the child elements (such as patient information, examination indicators, and the like) in the root element, and then output the tag name and text content of each child element in the metadata file, such as "tag: {child.tag}, content: {child.text}".

[0306] Step S2302, field filling of the metadata content based on the rule-matching dynamic template, to obtain first description information.

[0307] In some embodiments, the field filling of the metadata content based on the rule-matching dynamic template can include: filling the dynamic template according to the metadata content, constructing a sentence description (i.e., the first description information), and automatically omitting invalid segments according to the presence or absence of the field.

[0308] For example, it is assumed that the metadata contains the following elements: age (Age), age category (Age Category), gender (Sex), height (Height), weight (Weight), respiratory condition (Respiratory_Condition), fever or muscle pain (Fever_or_Muscle_Pain).

[0309] The template description is: the patient is a ({age}-year-old) ({age category}) ({gender}), (height {height} cm) and (weight {weight} kg) (suffering from respiratory condition) and (having fever or muscle pain).

[0310] Based on the above metadata content and the dynamic template, the following first description information can be obtained: the patient is a 33-year-old adult female, 170 cm tall and 50 kg, suffering from respiratory condition and accompanied by fever or muscle pain.

[0311] If the metadata content has a default element relative to the dynamic template, the corresponding first description information can be as follows: the patient is a child male with fever or muscle pain.

[0312] Step S2303, the encoding end sends the first description information to the decoding end.

[0313] At step S2304, the decoding end determines the text feature according to the first description information.

[0314] In some embodiments, the decoding end can determine the text feature according to the third description information by using a preset algorithm or a pre-trained AI model.

[0315] In some embodiments, the text feature is determined based on the first weight matrix and the first low-rank matrix, the first low-rank matrix being trained based on the low-rank adaptation on the first initial low-rank matrix, and the first low-rank matrix being used to adjust the parameters of the first weight matrix.

[0316] Optionally, the parameters of the first weight matrix are unchanged during the training process. Optionally, the first initial low-rank matrix can be a matrix without training or with default parameters.

[0317] In some embodiments, the decoding end is deployed with a text feature extraction model.

[0318] In some embodiments, the decoding end calls the text feature extraction model through a preset interface.

[0319] In some embodiments, the text feature extraction model is used to extract the text feature from the third description information.

[0320] In some embodiments, the text feature extraction model can be, for example, a RoBERTa text model.

[0321] In some embodiments, the text feature extraction model can be based on a Transformer encoder, which is composed of multiple encoding layers, each of which is based on a self-attention mechanism to calculate the output representation of each word, and a feedforward neural network further processes the output of the self-attention layer. The hidden state of the last layer of the output encoder is taken as the vector at the first position of the sequence as the text feature representation.

[0322] For example, the text feature extraction model can be represented as: z t = f t (x t ); where x t represents the third description information, f t (·) represents the text feature extraction module, and z t is the text feature representation. Optionally, the dimension of the text feature can be 768.

[0323] In some embodiments, the text feature extraction model can be composed of the first weight matrix and the first low-rank matrix.

[0324] In some embodiments, the text feature extraction model includes the first weight matrix and the first low-rank matrix.

[0325] Exemplarily, similar to the audio feature extraction model, if the first weight matrix is denoted as W ∈ R h×k , two lightweight trainable low-rank matrices such as A ∈ R r×k , B ∈ R h×r (i.e., the first low-rank matrix) can be introduced to approximate the weight update. During training, the first weight matrix remains frozen, and only A and B are updated, reducing the trainable parameter amount. Wherein, the forward propagation process can be denoted as F = Wx + ΔWx = Wx + BAx, ΔWx is the adjustment amount for the first weight matrix, which is equal to the adjustment amount BAx for the first low-rank matrix, and F can represent the trained text feature extraction model.

[0326] In some embodiments, refer to other embodiments described before or after the description of the present embodiment for each step and its optional implementation, and other related parts of the description, which will not be repeated here.

[0327] Figure 3A is a flowchart of a data processing method according to an embodiment of the present disclosure. As shown in Figure 3A , the present embodiment relates to a data processing method, which is performed by a decoding end, and the above method comprises:

[0328] Step S3101, receiving compressed audio data sent by an encoding end.

[0329] Step S3102, obtaining audio features according to the compressed audio data, the compressed audio data being obtained by compressing original audio data of a target object by the encoding end.

[0330] Step S3103, determining an analysis result corresponding to the target object based on the audio features, text features, and a classification model, the analysis result being used to indicate classification information of a health state of the target object.

[0331] In some embodiments, determining the analysis result corresponding to the target object based on the audio features, text features, and the classification model comprises:

[0332] obtaining fusion features according to the audio features and the text features;

[0333] processing the fusion features based on a pre-trained classification model to determine the analysis result corresponding to the target object.

[0334] In some embodiments, the method further comprises:

[0335] receiving first information sent by the encoding end;

[0336] determining text features based on the first information;

[0337] Wherein, the first information comprises any of the following:

[0338] text feature;

[0339] metadata file;

[0340] first description information corresponding to the metadata file, the first description information being used for describing content in the metadata file based on a natural language.

[0341] In some embodiments, the first information includes the metadata file;

[0342] determining the text feature based on the first information includes:

[0343] generating second description information corresponding to the metadata file according to the metadata file, the second description information being used for describing content in the metadata file based on a natural language;

[0344] determining the text feature according to the second description information.

[0345] In some embodiments, the generating the second description information corresponding to the metadata file according to the metadata file includes:

[0346] parsing the metadata file to obtain metadata content;

[0347] converting the metadata content into a key-value pair prompt;

[0348] generating the second description information according to the key-value pair prompt.

[0349] In some embodiments, the first description information is obtained by filling fields of metadata content based on a dynamic template matched by a rule at an encoding end, and the metadata content is obtained by parsing the metadata file at the encoding end.

[0350] In some embodiments, the text feature is determined based on a first weight matrix and a first low-rank matrix, the first low-rank matrix being trained based on low-rank adaptation on a first initial low-rank matrix, and the first low-rank matrix being used for adjusting parameters of the first weight matrix.

[0351] In some embodiments, the compressed audio data is obtained by quantizing a first feature representation, and the first feature representation is obtained by encoding original audio data.

[0352] In some embodiments, the audio feature is obtained according to the compressed audio data, including:

[0353] determining restored audio data according to the compressed audio data;

[0354] determining the audio feature according to the restored audio data;

[0355] wherein the restored audio data is obtained by decoding and reconstructing the compressed audio data.

[0356] In some embodiments, the audio feature is determined based on a second weight matrix and a second low-rank matrix, the second low-rank matrix is trained based on a low-rank adaptation on a second initial low-rank matrix, and the second low-rank matrix is used to adjust parameters of the second weight matrix.

[0357] In some embodiments, the fusion feature is determined based on the audio feature and the text feature, including:

[0358] Based on the cross-attention mechanism, a query matrix is determined based on a third weight matrix corresponding to the audio feature, a key matrix is determined based on a fourth weight matrix corresponding to the text feature, and a value matrix is determined based on a fifth weight matrix corresponding to the text feature;

[0359] A cross-attention result is determined based on the query matrix, the key matrix, and the value matrix.

[0360] The cross-attention result is fused with the audio feature and the text feature to obtain the fusion feature.

[0361] In some embodiments, the classification model includes a Transformer encoder and a classification head, and the analysis result corresponding to the target object is determined based on the fusion feature, including:

[0362] Based on the Transformer encoder, an encoded feature is determined based on the fusion feature.

[0363] Based on the classification head, the analysis result is determined based on the encoded feature.

[0364] In some embodiments, at least one of the first low-rank matrix, the second low-rank matrix, the third weight matrix, the fourth weight matrix, the fifth weight matrix, and the classification model is trained based on a loss function.

[0365] In some embodiments, the computing resources of the encoding end are higher than the computing resources of the encoding end.

[0366] In some embodiments, reference can be made to the steps and their optional implementation manners in other embodiments described before or after the description of the present embodiment, and other related parts of the description, which will not be repeated here.

[0367] Figure 3B is a flowchart of a data processing method according to an embodiment of the present disclosure. As shown in Figure 3B The present disclosure relates to a data processing method, which is performed by an encoding end, and the above method includes:

[0368] In step S3201, the original audio data and the metadata file of the target object are obtained.

[0369] In step S3202, the original audio data is compressed to obtain compressed audio data.

[0370] Step S3203, sending the compressed audio data to the decoding end, the compressed audio data being used for the decoding end to determine the audio feature.

[0371] Step S3204, sending the first information to the decoding end, the first information being used for the decoding end to determine the text feature.

[0372] Step S3205, receiving the analysis result of the target object sent by the decoding end, the analysis result being used for indicating the classification information of the health state of the target object, the analysis result being determined based on the audio feature, the text feature and the classification model.

[0373] In some embodiments, the first information comprises any of the following:

[0374] the text feature;

[0375] the metadata file;

[0376] first description information corresponding to the metadata file, the first description information being used for describing the content in the metadata file based on a natural language.

[0377] In some embodiments, the method further comprises:

[0378] parsing the metadata file to obtain metadata content;

[0379] performing field filling on the metadata content in a manner of a dynamic template based on rule matching to obtain the first description information.

[0380] In some embodiments, the method further comprises:

[0381] parsing the metadata file to obtain metadata content;

[0382] converting the metadata content into a key-value pair prompt;

[0383] determining third description information corresponding to the metadata file according to the key-value pair prompt;

[0384] determining the text feature according to the third description information.

[0385] In some embodiments, the compressed audio data is obtained by quantizing a first feature representation, the first feature representation being obtained by encoding original audio data.

[0386] In some embodiments, the computing resource of the decoding end is higher than that of the encoding end.

[0387] In some embodiments, refer to other embodiments described before or after the description corresponding to the present embodiment, the steps and optional implementation manners thereof, and other related parts in the description will not be repeated here.

[0388] Figure 3C is an interaction schematic diagram of a data processing method according to an embodiment of the present disclosure. As shown in Figure 3C the present embodiment relates to a data processing method, and the method comprises:

[0389] In step S3301, the encoding end compresses the original audio data to obtain compressed audio data.

[0390] In step S3302, the encoding end sends the compressed audio data and the metadata file to the decoding end.

[0391] In step S3303, the decoding end obtains audio features according to the compressed audio data.

[0392] In step S3304, the decoding end determines text features according to the metadata file.

[0393] In step S3305, the decoding end determines an analysis result corresponding to the target object based on the audio features, the text features, and a classification model.

[0394] In some embodiments, see the steps and their optional implementation manners in other embodiments described before or after the description of the present embodiment, and other related parts in the description, which will not be repeated here.

[0395] Figure 3D is an interaction schematic diagram of a data processing method according to an embodiment of the present disclosure. As shown in Figure 3D the present embodiment relates to a data processing method, and the method comprises:

[0396] In step S3401, the encoding end compresses the original audio data to obtain compressed audio data.

[0397] In step S3402, the encoding end determines first description information according to the metadata file.

[0398] In step S3403, the encoding end sends the compressed audio data and the first description information to the decoding end.

[0399] In step S3404, the decoding end obtains audio features according to the compressed audio data.

[0400] In step S3405, the decoding end determines text features according to the first description information.

[0401] In step S3406, the decoding end determines an analysis result corresponding to the target object based on the audio features, the text features, and a classification model.

[0402] In some embodiments, refer to the other embodiments described before or after the corresponding description of the present embodiment, the steps and their optional implementations, and other related parts of the description will not be repeated here.

[0403] Figure 3E is a schematic diagram of a data processing method according to an embodiment of the present disclosure. As shown in Figure 3E the present disclosure relates to a data processing method, the method comprising:

[0404] Step S3501, the encoding end compresses the original audio data to obtain compressed audio data.

[0405] Step S3502, the encoding end determines the text feature according to the metadata file.

[0406] Step S3503, the encoding end sends the compressed audio data and the text feature to the decoding end.

[0407] Step S3504, the decoding end obtains the audio feature according to the compressed audio data.

[0408] Step S3505, the decoding end determines the analysis result corresponding to the target object based on the audio feature, the text feature and the classification model.

[0409] In some embodiments, refer to the other embodiments described before or after the corresponding description of the present embodiment, the steps and their optional implementations, and other related parts of the description will not be repeated here.

[0410] Figure 4A is a schematic diagram of a data processing system according to an embodiment of the present disclosure. As shown in Figure 4A the system can include a decoding end 101 and an encoding end 102, and at least one of the following modules can be deployed in the decoding end 101 and the encoding end 102:

[0411] Audio compression module 401: used for high-fidelity compression of input audio (such as original audio data);

[0412] Audio decompression module 402: used for reconstructing audio from compressed audio signal (such as compressed audio data);

[0413] Audio feature extraction module 403: used for extracting complex semantic features (such as audio features) from decompressed audio (such as restored audio data);

[0414] First metadata processing module 404: used for converting XML format metadata into sentence description through dynamic template;

[0415] Second metadata processing module 405: used for converting XML format metadata into sentence description through language model;

[0416] Text feature extraction module 406: used for converting XML format metadata into sentence description and extracting features of metadata text (such as text features);

[0417] Feature fusion module 407: deep fusion of audio features and metadata text features based on cross-attention mechanism;

[0418] Classifier module 408: health status analysis (such as obtaining analysis results) according to fusion features.

[0419] In some embodiments, the above-mentioned modules can be computer-implemented software functional modules, for example, the audio compression module 401 can be implemented based on an audio encoder or an audio compression model, the audio feature extraction module 403 can be implemented based on a first weight matrix and a first low-rank matrix or based on an audio feature extraction model, the classifier module 408 can be implemented based on a classification model, and so on.

[0420] In some embodiments, the first metadata processing module 404 and the first metadata processing module 404 can be used alternatively according to system environment and computing resource conditions; the text feature extraction module 406 can also be deployed at the decoding end 101 or the encoding end 102 according to system environment and computing resource conditions.

[0421] In some embodiments, the audio compression module 401 can have the following features:

[0422] The original audio is input into the pre-trained audio encoder WavTokenizer-Encoder to generate a latent feature representation of the audio signal through a fully convolutional encoder network; then input into a single quantizer for quantization to generate a discretized feature representation, realizing efficient compression of the audio signal.

[0423] In some embodiments, the audio decompression module 402 can have the following features:

[0424] The compressed signal sent by the encoding end 102 is decoded by the pre-trained audio decoder WavTokenizer-Decoder to reconstruct an audio signal close to the original quality.

[0425] In some embodiments, the audio feature extraction module 403 can have the following features:

[0426] The audio is input into the pre-trained acoustic model Whisper to extract audio features.

[0427] In some embodiments, the first metadata processing module 404 can have the following features:

[0428] parsing the metadata file in XML format to extract the content of the metadata;

[0429] The metadata content is converted into a sentence description by using a dynamic template method based on rule matching. A sentence dynamic template is set in advance, and the corresponding fields in the template are filled according to the metadata content to construct the corresponding sentence description.

[0430] In this way, the compression of the XML metadata file can be completed at the encoding end 102 to reduce the data transmission load and improve the transmission efficiency.

[0431] In some embodiments, the second metadata processing module 405 includes the following features:

[0432] parsing the metadata file in XML format to extract the content of the metadata;

[0433] The metadata content is converted into a sentence description by using a generation method based on the pre-trained language model BioGPT. The metadata content is converted into a key-value pair prompt and input into the language model BioGPT to automatically generate a sentence description.

[0434] In this way, the language model is deployed at the decoding end 101 to process the XML metadata file transmitted by the encoding end 102, which can more flexibly adapt to different types and structures of metadata and improve the processing capacity and intelligent level of the system for diversified metadata.

[0435] In some embodiments, the text feature extraction module 406 includes the following features:

[0436] The sentence description of the metadata is input into the pre-trained text model RoBERTa to extract text features.

[0437] In some embodiments, the text feature extraction module 406 can be deployed at the decoding end 101 or the encoding end 102 according to the system environment and computing resource situation. If the computing resources of the encoding end 102 are sufficient, the text feature extraction is completed at the encoding end 102, which can further compress the metadata and reduce the data load transmitted to the decoding end 101, thereby improving the overall communication efficiency of the system. If the computing resources of the encoding end 102 are limited, the text feature extraction is performed at the decoding end 101.

[0438] In some embodiments, the feature fusion module 407 can have the following features:

[0439] Based on the cross-attention mechanism, the audio feature representation is linearly transformed to obtain a query matrix Q, and the text feature representation is linearly transformed to obtain a key matrix K and a value matrix V:

[0440] The attention score is calculated and weighted summed to obtain the first fusion feature z based on the cross-attention mechanism att:

[0441]

[0442] where d k is the attention dimension. for stabilizing the gradient.

[0443] The cross-attention result is spliced with the original audio feature z a and the text feature z t , and finally the fusion feature z is obtained: z = concat(z att , z a , z t ).

[0444] The fusion feature generated based on the attention mechanism realizes the deep fusion of the audio feature and the text feature, and splicing the original features to enhance information retention.

[0445] In some embodiments, the classifier module 408 includes the following features:

[0446] The fusion feature is processed by the Transformer encoder to generate a feature representation, and then input to the classification head:

[0447] p = Classifier (Transformer (z));

[0448] where p is the probability distribution of the output category generated by the classification head, and the category with the highest probability is the health status diagnosis result.

[0449] In some embodiments, the training of each module in the above system includes:

[0450] The LoRA method is used to fine-tune the pre-trained acoustic base model and the text base model, update the weight matrix in the cross-attention mechanism in the feature fusion, and update the parameters of the classifier. The cross-entropy loss function is used as the training target:

[0451]

[0452] where N is the number of samples, C is the number of categories, y i represents the true label of sample i, and p i represents the predicted probability of sample i in category c.

[0453] In some embodiments, with reference to Figure 4AThe system shown, the system can take audio data and metadata file as input, the audio signal is compressed by the audio compression module 401 at the encoding end 102; the compressed signal is transmitted to the decoding end 101, the compressed audio is recovered by the audio decompression module 402, and the audio feature representation is generated by the audio feature extraction module 403;

[0454] Referring to Figure 4A , the processing of the metadata provides three optional schemes, and in the implementation process, one of the three schemes can be selected according to system resources and scene requirements:

[0455] In the first mode, the XML format metadata is processed at the encoding end 102 by the first metadata processing module 404, the metadata sentence description is transmitted to the decoding end 101, and the text metadata feature representation is obtained by the text feature extraction module 406;

[0456] In the second mode, the XML format metadata is processed at the encoding end 102 by the first metadata processing module 404, and the text metadata feature representation is obtained by the text feature extraction module 406, and then the feature representation is transmitted to the decoding end 101;

[0457] In the third mode, the XML format metadata is processed at the decoding end 101 by the second metadata processing module 405, and then the text metadata feature representation is obtained by the text feature extraction module 406.

[0458] Among them, the audio feature and the text feature pass through the feature fusion module 407, and finally the analysis result of the health state is obtained through the classifier module 408.

[0459] In a specific embodiment, the audio compression module 401 is deployed at the encoding end 102, and is used for compressing the input audio to generate a discrete representation for transmission.

[0460] Optionally, the audio compression module 401 includes a preprocessing unit, which is used for resampling the health-related audio data to 24kHz.

[0461] Optionally, the audio compression module 401 includes an encoder, which is used for inputting the original health-related audio into a pre-trained audio encoder WavTokenizer-Encoder, down-sampling 320 times by a fully convolutional network encoder, generating a latent feature representation of the audio, and using a quantizer with a codebook space of 2^12 to discretize and compress the feature representation.

[0462] In a specific embodiment, the audio decompression module 402 is deployed in the decoding end 101, and is configured to input the compressed signal transmitted by the encoding end 102 into the pre-trained audio decoder WavTokenizer-Decoder, and reconstruct the audio signal close to the original audio quality through the Conv1D layer, the attention module and the ConvNeXt module.

[0463] In a specific embodiment, the audio feature extraction module 403 is deployed in the decoding end 101, and is configured to extract the audio features by using the pre-trained acoustic model.

[0464] Optionally, the audio feature extraction module 403 extracts the audio features by using the Whisper model.

[0465] In the decoding end 101, the health-related audio can be input into the pre-trained acoustic model Whisper. The Whisper adopts an encoder-decoder architecture. The encoder converts the audio signal into a mel-spectrogram, and then passes it through multiple layers of Transformer encoders to capture the long-distance dependency of the audio sequence. The decoder generates a text sequence based on the output of the encoder through the cross-attention mechanism. The output encoder hidden state takes the last layer hidden state as the audio feature representation, and is mapped to a representation space with a dimension of 768 through a linear projection layer: z a =Projection(f a (x a ));

[0466] In the decoding end 101, the health-related audio can be input into the pre-trained acoustic model Whisper. The Whisper adopts an encoder-decoder architecture. The encoder converts the audio signal into a mel-spectrogram, and then passes it through multiple layers of Transformer encoders to capture the long-distance dependency of the audio sequence. The decoder generates a text sequence based on the output of the encoder through the cross-attention mechanism. The output encoder hidden state takes the last layer hidden state as the audio feature representation, and is mapped to a representation space with a dimension of 768 through a linear projection layer: z a a a

[0467] In a specific embodiment, the first metadata processing module 404 is deployed in the encoding end 102, and is configured to convert the XML format metadata into a sentence description.

[0468] Optionally, the first metadata processing module 404 includes a preprocessing unit configured to match the corresponding XML format metadata file according to the audio file name.

[0469] Optionally, the encoding end 102 can parse the XML by using the xml.etree.ElementTree of Python, and extract the tag content.

[0470] Optionally, the encoding end 102 can convert the metadata content into a sentence description form.

[0471] ​​​For example, a dynamic template method based on rule matching is adopted, a sentence dynamic template is preset, the dynamic template is filled according to metadata content, a sentence description is constructed, and invalid fragments are automatically omitted according to the presence or absence of fields.

[0472] Suppose the metadata contains the following elements: Age, Age Category, Sex, Height, Weight, Respiratory_Condition, Fever_or_Muscle_Pain.

[0473] Template description: The Patient is a({Age}-year-old)({Age Category})({Sex}),({Height}cm tall)and(weighing{Weight}kg)(with respiratory_condition)and(feveror muscle pain).

[0474] In a complete metadata example, the sentence description is: ``The Patient is a 33-year-old adult female, 170cm tall and weighing 50kg, with a respiratory condition and fever or muscle pain.``; in a partial metadata instance, the sentence description is: ``The Patient is a child man with fever or muscle pain.``

[0475] In a specific embodiment, the text feature extraction module 406 adopts a pre-trained text model RoBERTa.

[0476] Optionally, the metadata sentence description is input to the pre-trained text model RoBERTa. RoBERTa is based on a Transformer encoder, which is composed of multiple encoding layers. Each encoding layer is based on a self-attention mechanism to calculate the output representation of each word, and a feedforward neural network further processes the output of the self-attention layer. The hidden state of the last layer of the output encoder is output, and the vector at the first position of the sequence is taken as the text feature representation:

[0477] z t =f t (x t );

[0478] wherein f t (·) represents a text base model, and z t768 for text feature representation.

[0479] Optionally, as Figure 4A mode one and mode three, the text feature extraction module 406 can be deployed at the decoding end 101. Optionally, as Figure 4A mode two, the text feature extraction module 406 can be deployed at the encoding end 102, which extracts features from the metadata sentence description, obtains low-dimensional representation, and then transmits to the decoding end 101, realizing further compression of metadata. The deployment location of the text feature extraction module 406 can be flexibly determined according to the system performance requirements of the end, to achieve a balance between computing efficiency and communication cost.

[0480] In a specific embodiment, the feature fusion module 407 is deployed at the decoding end 101, which can refer to Figure 4B as shown, and mainly realizes feature fusion based on cross-attention mechanism, with the specific steps as follows: Figure 4B

[0481] The audio feature representation is transformed by linear transformation to obtain the query matrix Q, and the text feature representation is transformed by linear transformation to obtain the key matrix K and the value matrix V:

[0482] Q = z a W Q ;

[0483] K = z t W K ;

[0484] V = z t W K ;

[0485] Wherein, W Q ,W K ,W V are trainable weight matrices.

[0486] Calculate the attention score and weighted sum to obtain the final first fusion feature z att based on cross-attention mechanism (i.e. cross-attention result):

[0487]

[0488] Wherein, d k is the attention dimension. Used to stabilize the gradient.

[0489] Concatenate the cross-attention result with the original audio feature and text feature to enhance information retention, and finally obtain the fusion feature: z = concat(z att ,z​a z t )。

[0490] In a specific embodiment, the classifier module 408 is deployed at the decoding end 101 and is mainly based on the Transformer structure.

[0491] Optionally, the classifier module 408 includes a Transformer encoder.

[0492] The input features can be first uniformly mapped to a 768-dimensional representation space by a linear projection layer: z' = Projection(z);

[0493] After processing by the Transformer encoder (12 attention heads, hidden layer dimension 768) and normalization by the LayerNorm layer, the vector at the first position in the sequence is taken as the global feature representation: y = LayerNorm(Transformer(z'))[0].

[0494] Optionally, the classifier module 408 further includes a classification head, which can employ a classification head composed of three fully connected layers, each layer introducing a ReLU activation function and a Dropout regularization operation with a probability of 0.5, and finally outputting the probability distribution of the class:

[0495] p = Classifier(y);

[0496] where p is the probability distribution of the output class generated by the classification head, and the class with the highest probability can be the health status diagnosis result.

[0497] In some embodiments, the training process of the health status diagnosis system based on audio and metadata fusion can optimize the performance of the audio feature extraction module 403, the text feature extraction module 406, the feature fusion module 407, and the classifier module 408.

[0498] The training process is based on a health-related audio dataset and corresponding metadata, and the goal is to fuse audio and text features to achieve high accuracy in health status diagnosis.

[0499] In a specific example, the following training preparation can be performed:

[0500] Dataset: KAUH lung sound dataset with sampling rate of 4000Hz, containing two classes of labels Asthma (Asthma and Asthmaand lung fibrosis) and Healthy (N); Circor heart sound dataset with sampling rate of 4000Hz, containing two classes of labels Normal and Abnormal; CoughVID cough sound dataset with sampling rate of 48000Hz, containing two classes of labels COVID-19 (COVID-19 and symptomatic) and Healthy. For the case of missing labels, the corresponding data is excluded. For the class-imbalance dataset (CoughVID dataset), the minority class data is augmented by random time-domain shifting and amplitude changing. The dataset is randomly divided into training set and test set in the ratio of 8:2.

[0501] Hardware environment: The training is performed on a computing platform equipped with NVIDIA 3090 GPU, and the batch size is set to 16.

[0502] Software framework: implemented using PyTorch 3.0, optimizer is Adam, initial learning rate is 5x10 -4 , weight decay is 1x10 -2 .

[0503] In some embodiments, the following training can be performed to obtain the trained audio feature extraction module 403, text feature extraction module 406, feature fusion module 407 and classifier module 408:

[0504] The pre-trained acoustic base model and text base model are fine-tuned using the LoRA method. For the weight matrix W ∈ R h×k of the pre-trained model, LoRA approximates the weight update by introducing two lightweight trainable low-rank matrices A ∈ R r×k , B ∈ R h×r .

[0505] During training, W remains frozen, and only A and B are updated, reducing the number of trainable parameters. The forward propagation process is: F = Wx + ΔWx = Wx + BAx.

[0506] Fine-tune the acoustic base model weights and text base model weights, and update the parameters of the classifier.

[0507] Training objective: minimize cross-entropy loss, loss function is:

[0508] where N is the number of samples, C is the number of classes, y i represents the true label of sample i, p i represents the predicted probability of sample i in class c.

[0509] In some embodiments, the loss function is calculated The low-rank matrices A and B in LoRA, the weight matrix in cross-attention mechanism and the classifier parameters are updated by back propagation.

[0510] In some embodiments, the hyperparameters can be set as the learning rate of 5x10 -4 , and the training is performed for 50 epochs.

[0511] In the embodiments of the present disclosure, at least one of the following advantages is achieved:

[0512] By fine-tuning the pre-trained acoustic model and text model, deep semantic features in audio and text are extracted respectively, which can more fully mine the semantic information inside the modal and provide more comprehensive and effective feature support for the classifier compared with the method relying on artificial feature engineering or shallow neural network.

[0513] Converting the metadata in XML format into natural language description helps to extract the semantic information of the metadata, has the ability to dynamically adapt to the missing of metadata and the change of fields, and enhances the adaptability of the system.

[0514] The first metadata processing module 404 realizes the compression of metadata on the encoding end 102 side, reduces the data transmission load, and improves the real-time response capability of the health diagnosis system.

[0515] The second metadata processing module 405 automatically generates sentence descriptions through a language model, further improving the flexibility of the system in processing different metadata content.

[0516] The cross-attention mechanism is used to realize the deep fusion of audio features and text features, which can capture more fine-grained correlation information between modalities compared with the traditional feature splicing and weighted fusion, thereby improving the classification accuracy.

[0517] In some embodiments, reference can be made to the steps and their optional implementation manners in other embodiments described before or after the description of the present embodiment, and other related parts in the description, which will not be repeated here.

[0518] The embodiments of the present disclosure also propose a device (which can also be referred to as a data processing device or a data processing equipment, etc.) for implementing any of the above methods. For example, a device is proposed, which includes units or modules for implementing the steps performed by the decoding end 101 in any of the above methods. For another example, another device is proposed, which includes units or modules for implementing the steps performed by the encoding end 102 in any of the above methods.

[0519] It should be understood that the division of each unit or module in the above apparatus is only a logical function division, and all or part of them can be integrated into a physical entity or physically separated in actual implementation. In addition, the units or modules in the apparatus can be implemented in the form of processor calling software: for example, the apparatus includes a processor connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit or module of the above apparatus, wherein the processor is, for example, a general processor such as a central processing unit (CPU) or a microprocessor, and the memory is a memory in the apparatus or a memory outside the apparatus. Alternatively, the units or modules in the apparatus can be implemented in the form of hardware circuit, and the functions of part or all of the units or modules can be realized by the design of hardware circuit. The above hardware circuit can be understood as one or more processors; for example, in one implementation, the above hardware circuit is an application-specific integrated circuit (ASIC), and the functions of part or all of the above units or modules are realized by the design of the logical relationship of elements in the circuit; for example, in another implementation, the above hardware circuit is a programmable logic device (PLD), and a field programmable gate array (FPGA) is taken as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of part or all of the above units or modules. All units or modules of the above apparatus can be implemented in the form of processor calling software, or all units or modules can be implemented in the form of hardware circuit, or part of the units or modules are implemented in the form of processor calling software, and the remaining part is implemented in the form of hardware circuit.

[0520] In the embodiments of the present disclosure, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), or the like. In another implementation, the processor can implement certain functions through a logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or can be reconfigured. For example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the configuration of the hardware circuit. It can be understood that the processor loads instructions to implement the functions of the above part or all units or modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), and the like.

[0521] Figure 5A FIG. 1 is a structural schematic diagram of a decoding end according to some embodiments of the present disclosure. The decoding end 101 is configured to execute any of the above methods. In some embodiments, as shown in FIG. 1, the decoding end 101 can include the following modules. Figure 5A

[0522] The first receiving module 5101 is configured to receive compressed audio data sent by the encoding end.

[0523] The feature extraction module 5102 is configured to obtain audio features according to the compressed audio data, the compressed audio data being obtained by compressing original audio data of the target object by the encoding end.

[0524] The analysis module 5103 is configured to determine an analysis result corresponding to the target object based on the audio features, the text features, and the classification model, the analysis result being used to indicate classification information of the health state of the target object.

[0525] In some embodiments, the analysis module 5103 is configured to:

[0526] ​According to the audio feature and the text feature, a fusion feature is obtained;

[0527] The fusion feature is processed based on a pre-trained classification model to determine an analysis result corresponding to the target object.

[0528] In some embodiments, the decoding end 101 further includes:

[0529] The second receiving module receives the first information sent by the encoding end;

[0530] The first determining module determines the text feature based on the first information;

[0531] The first information includes any of the following:

[0532] The text feature;

[0533] The metadata file;

[0534] The first description information corresponding to the metadata file, the first description information being used to describe the content in the metadata file based on a natural language.

[0535] In some embodiments, the first information includes the metadata file;

[0536] The first determining module is configured to:

[0537] generate second description information corresponding to the metadata file according to the metadata file, the second description information being used to describe the content in the metadata file based on a natural language;

[0538] determine the text feature according to the second description information.

[0539] In some embodiments, the first determining module is further configured to:

[0540] parse the metadata file to obtain metadata content;

[0541] convert the metadata content into a key-value pair prompt;

[0542] generate the second description information according to the key-value pair prompt.

[0543] In some embodiments, the first description information is obtained by filling fields of the metadata content based on a dynamic template matched by the encoding end, and the metadata content is obtained by parsing the metadata file by the encoding end.

[0544] In some embodiments, the text feature is determined based on a first weight matrix and a first low-rank matrix, the first low-rank matrix being trained based on low-rank adaptation on a first initial low-rank matrix, and the first low-rank matrix being used to adjust parameters of the first weight matrix.

[0545] In some embodiments, the compressed audio data is quantized from a first feature representation, the first feature representation is encoded from the original audio data.

[0546] In some embodiments, the feature extraction module 5102 is configured to:

[0547] determine the restored audio data according to the compressed audio data;

[0548] determine the audio feature according to the restored audio data;

[0549] wherein the restored audio data is reconstructed from the decoding of the compressed audio data.

[0550] In some embodiments, the audio feature is determined based on a second weight matrix and a second low-rank matrix, the second low-rank matrix is trained from the second initial low-rank matrix based on the low-rank adaptation, and the second low-rank matrix is used to adjust the parameters of the second weight matrix.

[0551] In some embodiments, the analysis module 5103 is further configured to:

[0552] determine a query matrix according to a third weight matrix corresponding to the audio feature based on the cross-attention mechanism, determine a key matrix according to a fourth weight matrix corresponding to the text feature, and determine a value matrix according to a fifth weight matrix corresponding to the text feature;

[0553] determine a cross-attention result according to the query matrix, the key matrix, and the value matrix;

[0554] fuse the cross-attention result with the audio feature and the text feature to obtain a fused feature.

[0555] In some embodiments, the classification model includes a Transformer encoder and a classification head, and the analysis module 5103 is further configured to:

[0556] determine an encoded feature according to the fused feature based on the Transformer encoder;

[0557] determine the analysis result according to the encoded feature based on the classification head.

[0558] In some embodiments, at least one of the first low-rank matrix, the second low-rank matrix, the third weight matrix, the fourth weight matrix, the fifth weight matrix, and the classification model is trained based on a loss function.

[0559] In some embodiments, the computing resource of the encoding end is higher than that of the decoding end.

[0560] In some embodiments, refer to other embodiments described before or after the corresponding description of the present embodiment for each step and its optional implementation, as well as other related parts of the description, which will not be repeated here.

[0561] Figure 5B is a structural schematic diagram of the encoding end 102 proposed in the embodiments of the present disclosure. The encoding end 102 is used to execute any of the above methods. In some embodiments, as shown in Figure 5B the encoding end can include:

[0562] The acquisition module 5201 is configured to acquire original audio data and a metadata file of a target object.

[0563] The compression module 5202 is configured to compress the original audio data to obtain compressed audio data.

[0564] The first sending module 5203 is configured to send the compressed audio data to a decoding end, and the compressed audio data is used by the decoding end to determine an audio feature.

[0565] The second sending module 5204 is configured to send first information to the decoding end, and the first information is used by the decoding end to determine a text feature.

[0566] The third receiving module 5205 is configured to receive an analysis result of the target object sent by the decoding end, and the analysis result is used to indicate classification information of a health state of the target object, and the analysis result is determined based on the audio feature, the text feature, and a classification model.

[0567] In some embodiments, the first information includes any of the following:

[0568] The text feature;

[0569] The metadata file;

[0570] First description information corresponding to the metadata file, and the first description information is used to describe the content in the metadata file based on a natural language.

[0571] In some embodiments, the encoding end 102 further includes:

[0572] The first parsing module is configured to parse the metadata file to obtain metadata content.

[0573] The first generation module is configured to perform field filling on the metadata content in a manner of a dynamic template matched based on a rule to obtain the first description information.

[0574] In some embodiments, the encoding end 102 further includes:

[0575] The second parsing module is configured to parse the metadata file to obtain metadata content.

[0576] The conversion module is used to convert metadata content into key-value pair hints;

[0577] The second generation module is used to determine the third description information corresponding to the metadata file based on key-value pair prompts;

[0578] The second determining module is used to determine text features based on the third descriptive information.

[0579] In some embodiments, the compressed audio data is obtained by quantizing a first feature representation, which is obtained by encoding the original audio data.

[0580] In some embodiments, the computing resources at the decoding end are higher than those at the encoding end.

[0581] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0582] Figure 6A This is a schematic diagram of the structure of the data processing device 6100 proposed in this embodiment. The data processing device 6100 can be any device, chip, chip system, or processor with data processing capabilities or computing resources. The data processing device 6100 can be used to implement the methods described in the above method embodiments, and for details, please refer to the description in the above method embodiments.

[0583] like Figure 6A As shown, the data processing device 6100 is used to execute any of the above methods. In some embodiments, the data processing device 6100 includes one or more processors 6101. The processor 6101 may be a general-purpose processor or a special-purpose processor, such as a baseband processor or a central processing unit. The baseband processor may be used to process communication protocols and communication data, and the central processing unit may be used to control communication devices (e.g., base stations, baseband chips, terminal devices, terminal device chips, DUs or CUs, etc.), execute programs, and process program data. Optionally, the data processing device 6100 is used to execute any of the above methods. Optionally, one or more processors 6101 are used to invoke instructions to cause the data processing device 6100 to execute any of the above methods.

[0584] In some embodiments, the data processing device 6100 further comprises one or more transceivers 6102. When the data processing device 6100 comprises one or more transceivers 6102, the transceiver 6102 performs at least one of the steps of sending and / or receiving in the above-mentioned methods, and the processor 6101 performs at least one of the other steps. In alternative embodiments, the transceiver can comprise a receiver and / or a transmitter, which can be separate or integrated together. Alternatively, the terms transceiver, transceiving unit, transceiver, transceiving circuit, interface circuit, interface, etc. can replace each other; the terms transmitter, transmitting unit, transmitter, transmitting circuit, etc. can replace each other; the terms receiver, receiving unit, receiver, receiving circuit, etc. can replace each other.

[0585] In some embodiments, the data processing device 6100 further comprises one or more memories 6103 for storing data and / or instructions. Alternatively, the one or more processors 6101 are configured to invoke the instructions stored in the memory 6103 to cause the data processing device 6100 to perform any of the above methods. Alternatively, all or part of the memory 6103 can also be outside the data processing device 6100. In alternative embodiments, the data processing device 6100 can comprise one or more interface circuits 6104. Alternatively, the interface circuit 6104 is connected to the memory 6102, and the interface circuit 6104 can be used to receive data and / or instructions from the memory 6102 or other devices, and can be used to send data and / or instructions to the memory 6102 or other devices. For example, the interface circuit 6104 can read the data and / or instructions stored in the memory 6102 and send the data and / or instructions to the processor 6101.

[0586] The data processing device 6100 described in the above embodiments can be a network device or a terminal, but the scope of the data processing device 6100 described in the present disclosure is not limited thereto, and the structure of the data processing device 6100 can not be limited by Figure 6A The data processing device can be a standalone device or can be part of a larger device. For example, the data processing device can be: (1) a standalone integrated circuit (IC), or chip, or chip system or subsystem; (2) a set of one or more ICs, which can optionally also include storage means for storing data, programs and / or instructions; (3) an ASIC, such as a Modem; (4) a module that can be embedded within other devices; (5) a receiver, terminal device, intelligent terminal device, cellular phone, wireless device, handset, mobile unit, car equipment, network device, cloud device, artificial intelligence device, etc.; (6) others, etc.

[0587] Figure 6Bis a structural schematic diagram of the chip 6200 proposed in embodiments of the present disclosure. For the case where the data processing device 6100 can be a chip or a chip system, reference can be made to Figure 6B The structural schematic diagram of the chip 6200 shown is not limited.

[0588] The chip 6200 includes one or more processors 6201. The chip 6200 is configured to execute any of the above methods.

[0589] In some embodiments, the chip 6200 further includes one or more interface circuits 6202. Optionally, the terms interface circuit, interface, transceiver pin, etc. can be replaced by each other. In some embodiments, the chip 6200 further includes one or more memories 6203 for storing data and / or instructions. Optionally, all or part of the memory 6203 can be outside the chip 6200. Optionally, the interface circuit 6202 is connected with the memory 6203, the interface circuit 6202 can be configured to receive data and / or instructions from the memory 6203 or other devices, and the interface circuit 6202 can be configured to send data and / or instructions to the memory 6203 or other devices. For example, the interface circuit 6202 can read data and / or instructions stored in the memory 6203 and send the data and / or instructions to the processor 6201.

[0590] In some embodiments, the interface circuit 6202 performs at least one of the sending and / or receiving steps in the above methods. The interface circuit 6202 performs the communication steps such as sending and / or receiving in the above methods, for example, means that the interface circuit 6202 performs the interaction of data and / or instructions between the processor 6201, the chip 6200, the memory 6203 or the transceiver device. In some embodiments, the processor 6201 performs at least one of the other steps.

[0591] The modules and / or devices described in each of the embodiments of the virtual device, the physical device, the chip, etc. can be combined or separated as appropriate. Optionally, part or all of the steps can also be performed by a plurality of modules and / or devices in cooperation, which is not limited here.

[0592] The present disclosure also proposes a storage medium, and the above storage medium stores instructions, when the above instructions run on the data processing device, the data processing device executes any of the above methods. Optionally, the above storage medium is an electronic storage medium. Optionally, the above storage medium is a computer readable storage medium, but is not limited to this, it can also be a storage medium readable by other devices. Optionally, the above storage medium can be a non-transitory storage medium, but is not limited to this, it can also be a transitory storage medium.

[0593] The present disclosure also provides a program product comprising a program and / or instructions which, when executed by a data processing device, cause the data processing device to perform any of the above methods. Optionally, the program product is a computer program product. Optionally, the program product is stored on a storage medium.

[0594] The present disclosure also provides a computer program which, when executed on a computer, causes the computer to perform any of the above methods.

Claims

1. A data processing method, characterized by, The method is executed by a decoding end, and the method comprises: receiving compressed audio data sent by an encoding end; determining an audio feature according to the compressed audio data, the compressed audio data being obtained by compressing original audio data of a target object by the encoding end; determining an analysis result corresponding to the target object based on the audio feature, a text feature, and a classification model, the analysis result being used to indicate classification information of a health state of the target object.

2. The method of claim 1, wherein, The method further comprises: receiving first information sent by the encoding end; determining the text feature based on the first information; 3. The method of claim 1, wherein, wherein the first information comprises any of the following: the text feature; a metadata file; first description information corresponding to the metadata file, the first description information being used to describe content in the metadata file based on a natural language. The first information comprises the metadata file. The method further comprises: generating second description information corresponding to the metadata file according to the metadata file, the second description information being used to describe content in the metadata file based on a natural language; 4. The method of claim 3, wherein, determining the text feature according to the second description information. The method further comprises: parsing the metadata file to obtain metadata content; converting the metadata content into a key-value pair prompt; 5. The method of claim 4, wherein, generating the second description information according to the key-value pair prompt. The first description information is obtained by filling a field of metadata content based on a dynamic template matched by the encoding end, the metadata content being obtained by parsing the metadata file by the encoding end. The text feature is determined based on a first weight matrix and a first low-rank matrix, the first low-rank matrix being trained based on low-rank adaptation to a first initial low-rank matrix, and the first low-rank matrix being used to adjust parameters of the first weight matrix. The compressed audio data is obtained by quantizing a first feature representation, the first feature representation being obtained by encoding the original audio data.

6. The method of claim 3, wherein, The method further comprises:

7. The method according to any one of claims 1 to 6, characterized in that, determining restored audio data according to the compressed audio data; 8. The method according to any one of claims 1 to 7, characterized in that, determining the audio feature according to the restored audio data; 9. The method according to any one of claims 1 to 8, characterized in that, wherein the restored audio data is obtained by decoding and reconstructing the compressed audio data. The audio feature is determined based on a second weight matrix and a second low-rank matrix, the second low-rank matrix being trained based on low-rank adaptation to a second initial low-rank matrix, and the second low-rank matrix being used to adjust parameters of the second weight matrix. The method further comprises: determining a fusion feature according to the audio feature and the text feature; 10. The method of claim 9, wherein, ​ 11. The method according to any one of claims 1 to 10, characterized in that, ​ determine a query matrix according to a third weight matrix corresponding to the audio feature, determine a key matrix according to a fourth weight matrix corresponding to the text feature, and determine a value matrix according to a fifth weight matrix corresponding to the text feature; determine a cross-attention result according to the query matrix, the key matrix, and the value matrix; fuse the cross-attention result with the audio feature and the text feature to obtain the fusion feature.

12. The method according to any one of claims 1 to 11, characterized in that, The classification model includes a Transformer encoder and a classification head, and the determination of the analysis result corresponding to the target object according to the fusion feature includes: determining an encoded feature according to the fusion feature based on the Transformer encoder; determining the analysis result according to the encoded feature based on the classification head.

13. The method according to any one of claims 1 to 12, characterized in that, At least one of the first low-rank matrix, the second low-rank matrix, the third weight matrix, the fourth weight matrix, the fifth weight matrix, and the classification model is obtained based on a loss function.

14. The method according to any one of claims 1 to 13, characterized in that, The computing resource of the encoding end is higher than that of the encoding end.

15. A data processing method, characterized by, The method is executed by the encoding end, and the method includes: obtaining original audio data and a metadata file of a target object; compressing the original audio data to obtain compressed audio data; sending the compressed audio data to a decoding end, the compressed audio data being used by the decoding end to determine an audio feature; sending first information to the decoding end, the first information being used by the decoding end to determine a text feature; receiving an analysis result of the target object sent by the decoding end, the analysis result being used to indicate classification information of a health state of the target object, and the analysis result being determined based on the audio feature, the text feature, and a classification model.

16. The method of claim 15, wherein, The first information includes any of the following: the text feature; the metadata file; first description information corresponding to the metadata file, the first description information being used to describe content in the metadata file based on a natural language.

17. The method according to claim 15 or 16, characterized in that, The method further includes: parsing the metadata file to obtain metadata content; performing field filling on the metadata content in a manner of a rule-matching dynamic template to obtain the first description information.

18. The method of claim 15 or 16, wherein, The method further includes: parsing the metadata file to obtain metadata content; converting the metadata content into a key-value pair prompt; determining third description information corresponding to the metadata file according to the key-value pair prompt; determining the text feature according to the third description information.

19. The method according to any one of claims 15-18, characterized in that, The compressed audio data is obtained by quantizing a first feature representation, and the first feature representation is obtained by encoding the original audio data.

20. The method according to any one of claims 15-19, characterized by, The computing resource of the decoding end is higher than that of the encoding end.

21. A decoding end, comprising: The decoding end is configured to execute the data processing method in any one of claims 1-14.

22. An encoding end, comprising: The encoding end is configured to execute the data processing method in any one of claims 15-20.

23. A data processing system, characterized by The system comprises a decoding side configured to implement the data processing method of any one of claims 1-14, and an encoding side configured to implement the data processing method of any one of claims 15-20.

24. A storage medium, the storage medium storing instructions, wherein, The instructions, when executed on a device, cause the device to perform the data processing method of any one of claims 1-14, or any one of claims 15-20.

25. A computer program product comprising computer programs and / or instructions, characterized in that, The computer program and / or the instructions, when executed on a device, implement the data processing method of any one of claims 1-14, or any one of claims 15-20.

Images