Bone joint lesion recognition method and device based on multi-modal information fusion

Abstract

The invention provides a bone joint lesion recognition method and device based on multi-modal information fusion, electronic equipment and a computer readable storage medium. The bone joint lesion recognition method based on multi-modal information fusion comprises the following steps: acquiring a bone joint part image and text information; wherein the text information comprises bone joint questionnaire information and historical case information, and the bone joint questionnaire information comprises patient identity information and bone joint function evaluation information; performing image coding layer feature extraction on the bone joint part image to obtain image feature information; performing text coding layer feature extraction on the text information to obtain text feature information; performing multi-modal information fusion on the image feature information and the text feature information to obtain fused feature information; and bone joint lesion recognition is carried out based on the fusion feature information. According to the embodiment of the invention, the osteoarthropathy can be identified more accurately.

Description

Technical Field

[0001] The present application relates to the field of bone and joint lesion identification, and in particular to a bone and joint lesion identification method, device, electronic device and computer-readable storage medium based on multimodal information fusion. Background Art

[0002] At present, the recognition of bone and joint lesions (hip joint, knee joint, etc.) is mainly based on images of the bone and joint parts. However, if lesions are recognized based solely on images, inaccurate recognition may occur.

[0003] Therefore, how to more accurately identify bone and joint lesions is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device and computer-readable storage medium for identifying bone and joint lesions based on multimodal information fusion, which can more accurately identify bone and joint lesions.

[0005] In a first aspect, the present application provides a method for identifying bone and joint lesions based on multimodal information fusion, comprising:

[0006] Obtaining bone and joint images and text information; wherein the text information includes bone and joint questionnaire information and historical case information, and the bone and joint questionnaire information includes patient identity information and bone and joint function assessment information;

[0007] Perform image coding layer feature extraction on bone joint images to obtain image feature information;

[0008] Perform text encoding layer feature extraction on the text information to obtain text feature information;

[0009] Perform multimodal information fusion on the image feature information and the text feature information to obtain fused feature information;

[0010] Bone and joint lesions are identified based on fusion feature information.

[0011] Optionally, image coding layer feature extraction is performed on the bone joint image to obtain image feature information, including:

[0012] Performing image preprocessing on the bone joint images; wherein the image preprocessing includes denoising, contrast enhancement and normalization; using Gaussian filtering or median filtering to denoise the bone joint images;

[0013] Perform image segmentation on the bone and joint images after image preprocessing;

[0014] Extract features from the bone and joint images after image segmentation to obtain feature maps;

[0015] Choose the appropriate machine learning model based on the task, including support vector machine, random forest, neural network;

[0016] Train the model using the training dataset;

[0017] Evaluate the performance of the model using the validation dataset;

[0018] The model performance was evaluated using accuracy, recall, and F1 score metrics.

[0019] Optionally, image segmentation is performed on the bone and joint images after image preprocessing, including:

[0020] Thresholds are set according to pixel intensity, and threshold segmentation is performed on the bone joint images after image preprocessing to separate the joint area from the background;

[0021] Use the Canny edge detector to detect edges and identify the boundaries of joints;

[0022] Starting from one or more seed points, the region is gradually expanded according to the predefined pixel intensity similarity to perform region growing;

[0023] Use dilation, erosion, opening, and closing operations to improve segmentation results.

[0024] Optionally, feature extraction is performed on the bone and joint images after image segmentation to obtain a feature map, including:

[0025] Extract shape features of joints, including area, perimeter, and shape index;

[0026] Use gray-level co-occurrence matrix and local binary pattern method to analyze the texture information of the joint area;

[0027] Calculate the geometric characteristics of joints, including joint angles and joint spacing;

[0028] Automatically extract features using deep learning models using convolutional neural networks.

[0029] Optionally, feature selection and feature encoding are performed on the feature map, including:

[0030] Use principal component analysis and linear discriminant analysis methods to reduce feature dimensions and improve computational efficiency;

[0031] Determine which features contribute most to a classification or regression task through statistical tests or model evaluation;

[0032] Convert continuous feature values ​​into discrete codes;

[0033] Convert the extracted features into fixed-length vectors for input into machine learning models.

[0034] Optionally, text encoding layer feature extraction is performed on the text information to obtain text feature information, including:

[0035] Perform text preprocessing on the text information; the text preprocessing includes: cleaning, which is used to remove noise in the text; word segmentation, which is used to divide the text into words or phrases; stop word removal, which is used to remove common and meaningless words; stemming or word form restoration, which is used to restore words to their basic form;

[0036] Perform word embedding on the text information after text preprocessing;

[0037] Perform text representation on the text information after word embedding; the text representation includes: sentence and paragraph representation, which is used to merge the representation of a single word into the representation of a sentence or paragraph using average pooling, maximum pooling or weighted average; context embedding, which is used to obtain context-sensitive representation of words using context-dependent word embedding models;

[0038] Perform feature selection and dimensionality reduction on the text information after text representation; feature selection is used to select the most informative features based on statistical tests or model weights; PCA, t-SNE, and UMAP methods are used to reduce the dimensionality of the feature space while retaining the information of the original data;

[0039] Serialize the text information after feature selection and dimensionality reduction; the serialization process includes: padding and truncation, which are used to pad or truncate fixed-length input; sliding window, which is used to extract local context information from text sequences using sliding window technology;

[0040] A deep learning model is used to extract features from the serialized text information to obtain text feature information; wherein the deep learning model includes: a convolutional neural network, which is used to extract local features using a convolutional layer; a recurrent neural network, which is used to process sequence data using an LSTM or GRU variant; and an attention mechanism, which is used to weight the importance of different parts using an attention mechanism.

[0041] Optionally, multimodal information fusion is performed on the image feature information and the text feature information to obtain fused feature information, including:

[0042] By means of a connection operation, the image feature information and the text feature information are merged to obtain first fused feature information;

[0043] The image feature information and the text feature information are fused through a weighted average or voting mechanism to obtain second fused feature information;

[0044] At the decision-making level, the first fused feature information and the second fused feature information are fused to obtain final fused feature information.

[0045] In a second aspect, an embodiment of the present application provides a device for identifying bone and joint lesions based on multimodal information fusion, the device comprising:

[0046] An information acquisition module is used to acquire bone and joint images and text information; wherein the text information includes bone and joint questionnaire information and historical case information, and the bone and joint questionnaire information includes patient identity information and bone and joint function assessment information;

[0047] An image coding layer feature extraction module is used to extract image coding layer features of bone joint images to obtain image feature information;

[0048] A text coding layer feature extraction module is used to extract text coding layer features from text information to obtain text feature information;

[0049] A multimodal information fusion module is used to perform multimodal information fusion on image feature information and text feature information to obtain fused feature information;

[0050] The bone and joint lesion recognition module is used to identify bone and joint lesions based on fusion feature information.

[0051] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0052] When the processor executes the computer program instructions, the method for identifying bone and joint lesions based on multimodal information fusion as shown in the first aspect is implemented.

[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the method for identifying bone and joint lesions based on multimodal information fusion as shown in the first aspect is implemented.

[0054] The bone and joint lesion identification method, device, electronic device and computer-readable storage medium based on multimodal information fusion of the embodiment of the present application can more accurately identify bone and joint lesions. The bone and joint lesion identification method based on multimodal information fusion includes: obtaining bone and joint part images and text information; wherein the text information includes bone and joint questionnaire information and historical case information, and the bone and joint questionnaire information includes patient identity information and bone and joint function assessment information; performing image coding layer feature extraction on the bone and joint part images to obtain image feature information; performing text coding layer feature extraction on the text information to obtain text feature information; performing multimodal information fusion on the image feature information and the text feature information to obtain fused feature information; and performing bone and joint lesion identification based on the fused feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 It is a flowchart of a method for identifying bone and joint lesions based on multimodal information fusion provided by an embodiment of the present application;

[0057] Figure 2 It is a structural schematic diagram of a bone and joint lesion identification device based on multimodal information fusion provided by an embodiment of the present application;

[0058] Figure 3 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0059] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0060] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0061] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, electronic device and computer-readable storage medium for identifying bone and joint lesions based on multimodal information fusion. The following first introduces the method for identifying bone and joint lesions based on multimodal information fusion provided by the embodiments of the present application.

[0062] Figure 1 FIG. 1 is a flow chart of a method for identifying bone and joint lesions based on multimodal information fusion provided by an embodiment of the present application. Figure 1 As shown, the bone and joint lesion recognition method based on multimodal information fusion includes:

[0063] S101, obtaining bone and joint images and text information; wherein the text information includes bone and joint questionnaire information and historical case information, and the bone and joint questionnaire information includes patient identity information and bone and joint function assessment information;

[0064] S102, extracting image coding layer features from the bone joint image to obtain image feature information;

[0065] S103, extracting text coding layer features from the text information to obtain text feature information;

[0066] S104, performing multimodal information fusion on the image feature information and the text feature information to obtain fused feature information;

[0067] S105. Identify bone and joint lesions based on the fused feature information.

[0068] In one embodiment, image coding layer feature extraction is performed on the bone joint part image to obtain image feature information, including:

[0069] Performing image preprocessing on the bone joint images; wherein the image preprocessing includes denoising, contrast enhancement and normalization; using Gaussian filtering or median filtering to denoise the bone joint images;

[0070] Perform image segmentation on the bone and joint images after image preprocessing;

[0071] Extract features from the bone and joint images after image segmentation to obtain feature maps;

[0072] Choose the appropriate machine learning model based on the task, including support vector machine, random forest, neural network;

[0073] Train the model using the training dataset;

[0074] Evaluate the performance of the model using the validation dataset;

[0075] The model performance was evaluated using accuracy, recall, and F1 score metrics.

[0076] In one embodiment, performing image segmentation on the bone joint part image after image preprocessing includes:

[0077] Thresholds are set according to pixel intensity, and threshold segmentation is performed on the bone joint images after image preprocessing to separate the joint area from the background;

[0078] Use the Canny edge detector to detect edges and identify the boundaries of joints;

[0079] Starting from one or more seed points, the region is gradually expanded according to the predefined pixel intensity similarity to perform region growing;

[0080] Use dilation, erosion, opening, and closing operations to improve segmentation results.

[0081] In one embodiment, feature extraction is performed on the bone joint part image after image segmentation to obtain a feature map, including:

[0082] Extract shape features of joints, including area, perimeter, and shape index;

[0083] Use gray-level co-occurrence matrix and local binary pattern method to analyze the texture information of the joint area;

[0084] Calculate the geometric characteristics of joints, including joint angles and joint spacing;

[0085] Automatically extract features using deep learning models using convolutional neural networks.

[0086] In one embodiment, feature selection and feature encoding are performed on the feature map, including:

[0087] Use principal component analysis and linear discriminant analysis methods to reduce feature dimensions and improve computational efficiency;

[0088] Determine which features contribute most to a classification or regression task through statistical tests or model evaluation;

[0089] Convert continuous feature values ​​into discrete codes;

[0090] Convert the extracted features into fixed-length vectors for input into machine learning models.

[0091] In one embodiment, text information is subjected to text coding layer feature extraction to obtain text feature information, including:

[0092] Perform text preprocessing on the text information; the text preprocessing includes: cleaning, which is used to remove noise in the text; word segmentation, which is used to divide the text into words or phrases; stop word removal, which is used to remove common and meaningless words; stemming or word form restoration, which is used to restore words to their basic form;

[0093] Perform word embedding on the text information after text preprocessing;

[0094] Perform text representation on the text information after word embedding; the text representation includes: sentence and paragraph representation, which is used to merge the representation of a single word into the representation of a sentence or paragraph using average pooling, maximum pooling or weighted average; context embedding, which is used to obtain context-sensitive representation of words using context-dependent word embedding models;

[0095] Perform feature selection and dimensionality reduction on the text information after text representation; feature selection is used to select the most informative features based on statistical tests or model weights; PCA, t-SNE, and UMAP methods are used to reduce the dimensionality of the feature space while retaining the information of the original data;

[0096] Serialize the text information after feature selection and dimensionality reduction; the serialization process includes: padding and truncation, which are used to pad or truncate fixed-length input; sliding window, which is used to extract local context information from text sequences using sliding window technology;

[0097] A deep learning model is used to extract features from the serialized text information to obtain text feature information; wherein the deep learning model includes: a convolutional neural network, which is used to extract local features using a convolutional layer; a recurrent neural network, which is used to process sequence data using an LSTM or GRU variant; and an attention mechanism, which is used to weight the importance of different parts using an attention mechanism.

[0098] In one embodiment, multimodal information fusion is performed on the image feature information and the text feature information to obtain fused feature information, including:

[0099] By means of a connection operation, the image feature information and the text feature information are merged to obtain first fused feature information;

[0100] The image feature information and the text feature information are fused through a weighted average or voting mechanism to obtain second fused feature information;

[0101] At the decision-making level, the first fused feature information and the second fused feature information are fused to obtain final fused feature information.

[0102] Figure 2 It is a structural schematic diagram of a bone and joint lesion identification device based on multimodal information fusion provided by an embodiment of the present application;

[0103] The bone and joint lesion identification device based on multimodal information fusion includes:

[0104] The information acquisition module 201 is used to acquire bone and joint images and text information; wherein the text information includes bone and joint questionnaire information and historical case information, and the bone and joint questionnaire information includes patient identity information and bone and joint function assessment information;

[0105] The image coding layer feature extraction module 202 is used to extract the image coding layer features of the bone joint part image to obtain image feature information;

[0106] The text coding layer feature extraction module 203 is used to extract text coding layer features from text information to obtain text feature information;

[0107] The multimodal information fusion module 204 is used to perform multimodal information fusion on the image feature information and the text feature information to obtain fused feature information;

[0108] The bone and joint lesion recognition module 205 is used to recognize bone and joint lesions based on the fusion feature information.

[0109] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown.

[0110] The electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0111] Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0112] The memory 302 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 302 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 302 may be inside or outside the electronic device. In a particular embodiment, the memory 302 may be a non-volatile solid-state memory.

[0113] In one embodiment, the memory 302 may be a read-only memory (ROM). In one embodiment, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0114] The processor 301 reads and executes the computer program instructions stored in the memory 302 to implement any one of the bone and joint lesion identification methods based on multimodal information fusion in the above embodiments.

[0115] In one example, the electronic device may further include a communication interface 303 and a bus 310. Figure 3 As shown, the processor 301, the memory 302, and the communication interface 303 are connected via a bus 310 and communicate with each other.

[0116] The communication interface 303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0117] Bus 310 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 310 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0118] In addition, in combination with the bone and joint lesion identification method based on multimodal information fusion in the above embodiments, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the bone and joint lesion identification methods based on multimodal information fusion in the above embodiments is implemented.

[0119] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0120] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0121] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0122] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0123] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for identifying bone and joint lesions based on multimodal information fusion, characterized in that: include: Obtaining bone and joint images and text information; wherein the text information includes bone and joint questionnaire information and historical case information, and the bone and joint questionnaire information includes patient identity information and bone and joint function assessment information; Perform image coding layer feature extraction on bone joint images to obtain image feature information; Perform text encoding layer feature extraction on the text information to obtain text feature information; Perform multimodal information fusion on the image feature information and the text feature information to obtain fused feature information; Bone and joint lesions are identified based on fusion feature information.

2. The method for identifying bone and joint lesions based on multimodal information fusion according to claim 1, characterized in that: Perform image coding layer feature extraction on bone joint images to obtain image feature information, including: Performing image preprocessing on the bone joint images; wherein the image preprocessing includes denoising, contrast enhancement and normalization; using Gaussian filtering or median filtering to denoise the bone joint images; Perform image segmentation on the bone and joint images after image preprocessing; Extract features from the bone and joint images after image segmentation to obtain feature maps; Choose the appropriate machine learning model based on the task, including support vector machine, random forest, neural network; Train the model using the training dataset; Evaluate the performance of the model using the validation dataset; The model performance was evaluated using accuracy, recall, and F1 score metrics.

3. The method for identifying bone and joint lesions based on multimodal information fusion according to claim 2, characterized in that: Perform image segmentation on the bone and joint images after image preprocessing, including: Thresholds are set according to pixel intensity, and threshold segmentation is performed on the bone joint images after image preprocessing to separate the joint area from the background; Use the Canny edge detector to detect edges and identify the boundaries of joints; Starting from one or more seed points, the region is gradually expanded according to the predefined pixel intensity similarity to perform region growing; Use dilation, erosion, opening, and closing operations to improve segmentation results.

4. The method for identifying bone and joint lesions based on multimodal information fusion according to claim 3 is characterized in that: The bone and joint images after image segmentation are subjected to feature extraction to obtain feature maps, including: Extract shape features of joints, including area, perimeter, and shape index; Use gray-level co-occurrence matrix and local binary pattern method to analyze the texture information of the joint area; Calculate the geometric characteristics of joints, including joint angles and joint spacing; Automatically extract features using deep learning models using convolutional neural networks.

5. The method for identifying bone and joint lesions based on multimodal information fusion according to claim 4 is characterized in that: Perform feature selection and feature encoding on the feature map, including: Use principal component analysis and linear discriminant analysis methods to reduce feature dimensions and improve computational efficiency; Determine which features contribute most to a classification or regression task through statistical tests or model evaluation; Convert continuous feature values ​​into discrete codes; Convert the extracted features into fixed-length vectors for input into machine learning models.

6. The method for identifying bone and joint lesions based on multimodal information fusion according to claim 5, characterized in that: Perform text encoding layer feature extraction on the text information to obtain text feature information, including: Perform text preprocessing on the text information; the text preprocessing includes: cleaning, which is used to remove noise in the text; word segmentation, which is used to divide the text into words or phrases; stop word removal, which is used to remove common and meaningless words; stemming or word form restoration, which is used to restore words to their basic form; Perform word embedding on the text information after text preprocessing; Perform text representation on the text information after word embedding; the text representation includes: sentence and paragraph representation, which is used to merge the representation of a single word into the representation of a sentence or paragraph using average pooling, maximum pooling or weighted average; context embedding, which is used to obtain context-sensitive representation of words using context-dependent word embedding models; Perform feature selection and dimensionality reduction on the text information after text representation; feature selection is used to select the most informative features based on statistical tests or model weights; PCA, t-SNE, and UMAP methods are used to reduce the dimensionality of the feature space while retaining the information of the original data; Serialize the text information after feature selection and dimensionality reduction; the serialization process includes: padding and truncation, which are used to pad or truncate fixed-length input; sliding window, which is used to extract local context information from text sequences using sliding window technology; A deep learning model is used to extract features from the serialized text information to obtain text feature information; wherein the deep learning model includes: a convolutional neural network, which is used to extract local features using a convolutional layer; a recurrent neural network, which is used to process sequence data using an LSTM or GRU variant; and an attention mechanism, which is used to weight the importance of different parts using an attention mechanism.

7. The method for identifying bone and joint lesions based on multimodal information fusion according to claim 6, characterized in that: Perform multimodal information fusion on the image feature information and text feature information to obtain fused feature information, including: By means of a connection operation, the image feature information and the text feature information are merged to obtain first fused feature information; The image feature information and the text feature information are fused through a weighted average or voting mechanism to obtain second fused feature information; At the decision-making level, the first fused feature information and the second fused feature information are fused to obtain final fused feature information.

8. A bone and joint lesion identification device based on multimodal information fusion, characterized in that: The device includes: An information acquisition module is used to acquire bone and joint images and text information; wherein the text information includes bone and joint questionnaire information and historical case information, and the bone and joint questionnaire information includes patient identity information and bone and joint function assessment information; An image coding layer feature extraction module is used to extract image coding layer features of bone joint images to obtain image feature information; A text coding layer feature extraction module is used to extract text coding layer features from text information to obtain text feature information; A multimodal information fusion module is used to perform multimodal information fusion on image feature information and text feature information to obtain fused feature information; The bone and joint lesion recognition module is used to identify bone and joint lesions based on fusion feature information.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for identifying bone and joint lesions based on multimodal information fusion as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the method for identifying bone and joint lesions based on multimodal information fusion as described in any one of claims 1 to 7 is implemented.

Images