Identification and evaluation method and system based on multi-modal data analysis

Abstract

The invention relates to the technical field of image data processing, and provides a recognition and evaluation method and system based on multi-modal data analysis, and the method comprises the steps: obtaining image data and sequence data through a data obtaining module, and guaranteeing the comprehensiveness and accuracy of the data; the attribute analysis module judges target existence through the image data, analyzes target attributes in the image data and sequence attributes in the sequence data, converts original multi-modal data into attributes which can be used for analysis, reduces the complexity of the multi-modal data, and improves the accuracy of analysis and judgment; the fusion recognition module fuses the target attribute and the sequence attribute to obtain a target sequence attribute so as to recognize the comprehensive state, and the comprehensive state can be reflected more comprehensively; and the recognition evaluation module compares the recognized comprehensive state with the actual comprehensive state to obtain a feedback signal, and optimizes the recognition of the comprehensive state and the judgment of the existence of the target according to the feedback signal, thereby improving the recognition accuracy and reliability, and being capable of adapting to the dynamic change of the comprehensive state.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to a recognition and evaluation method and system based on multimodal data analysis. Background Art

[0002] Multimodal data analysis refers to the fusion of data from different modalities for analysis to obtain more comprehensive and accurate information. This analysis method goes beyond the limitations of traditional single-modal data and improves the accuracy and efficiency of data analysis by combining the advantages of each modal data. Multimodal data analysis has shown great potential in many fields, such as autonomous driving systems using multiple modal data such as images, radars, and lidars to achieve environmental perception, and the medical and health field improves diagnostic accuracy and treatment efficiency by integrating multimodal data such as CT and genetic data. However, most of them have not solved how to determine the existence of the target by obtaining image data and sequence data to identify the comprehensive state, and optimize the method of determining the existence of the target based on the feedback of the comprehensive state.

[0003] For example, a Chinese patent with authorization announcement number CN118762001B discloses a flexible circuit board quality inspection system and method based on data analysis, which obtains a reference image of the flexible circuit board, obtains the relative position relationship between the bending axis of the flexible circuit board and the position of the electronic components in the flexible circuit board from the reference image, compares the images of the flexible circuit board before and after the bending test, calculates the offset vector of the position of the electronic components in the image, obtains the inspection record of the flexible circuit board after the bending test, constructs a model of image changes regarding the quality of the flexible circuit board, generates an evaluation value of the failure rate of the flexible circuit board based on image recognition, reduces the amount of manual inspection of the flexible circuit board, and uses the relationship model between the offset vector and the failure rate, takes a certain circuit board in the flexible circuit board as the target circuit board, obtains images of the target circuit board before and after the bending test, and obtains the evaluation result of the quality of the target circuit board.

[0004] For example, the Chinese patent with the authorization announcement number CN115439452B discloses a capacitor product detection and evaluation system based on data analysis, including a first image acquisition module, etc. Through the set packaging defect detection module, the packaging defects of capacitor products can be easily detected, and the packaging defect results of capacitor products can be accurately obtained; through the set structural defect detection module, the structural defects of capacitor products can be easily detected, and whether the capacitor pins are missing or bent can be accurately determined; through the set defect analysis module, the quality of capacitors produced by this capacitor model in a unit time can be easily scored according to the packaging shell defect identification results, pin missing defect identification results, and pin bending defect identification results.

[0005] The above-mentioned prior art has the problem raised by the background technology: the above-mentioned prior art does not solve the problem of how to determine the existence of the target by acquiring image data and sequence data to identify the comprehensive state, and optimize the method of determining the existence of the target according to the feedback of the comprehensive state. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present application provides an identification and evaluation method and system based on multimodal data analysis.

[0007] In a first aspect, the present application provides a recognition and evaluation system based on multimodal data analysis, the system comprising: a data acquisition module, an attribute analysis module, a fusion recognition module and a recognition and evaluation module;

[0008] The data acquisition module is used to acquire image data and sequence data through a data acquisition strategy;

[0009] The attribute analysis module is used to judge the existence of the target through the image data, and analyze the target attributes in the image data and the sequence attributes in the sequence data. The judgment logic of the target existence includes capturing the target structure and expanding the target structure to obtain the target candidate area, and judging the existence of the target according to the gradient strength of the target candidate area;

[0010] The fusion recognition module is used to fuse the target attribute and the sequence attribute to obtain the target sequence attribute to identify the comprehensive state;

[0011] The identification and evaluation module is used to compare the identified comprehensive state with the actual comprehensive state to obtain a feedback signal, and optimize the identification of the comprehensive state and the judgment of the target existence according to the feedback signal. The judgment logic of optimizing the target existence includes dynamically adjusting the similarity threshold.

[0012] As an optional implementation, the data acquisition strategy includes:

[0013] Adjust the equipment's parameter configuration according to the target's detection requirements, calibrate the equipment's focus method, obtain image data and sequence data through the equipment, and control the consistency of image data and sequence data.

[0014] As an optional implementation, the logic for controlling the consistency of the image data and the sequence data includes:

[0015] Assigning timestamps to the image data and the sequence data, matching the timestamp of the image data with the timestamp of the sequence data, and if there is a time deviation between the timestamp of the image data and the timestamp of the sequence data, determining the degree of the time deviation;

[0016] The time difference is obtained by subtracting the timestamp of the image data from the timestamp of the sequence data, and a difference threshold is configured. If the time difference is less than or equal to the difference threshold, the degree of time deviation is the degree base value, and the time deviation is corrected by interpolation calculation;

[0017] If the time difference is greater than the difference threshold, the degree of the time deviation is the extreme value, and the time deviation is corrected by weighted average calculation.

[0018] As an optional implementation, the target existence judgment logic includes:

[0019] The target structure is captured by a convolutional neural network, and a target structure graph is generated. Each pixel in the target structure graph is used as a seed, and the target structure is expanded along the similarity between pixels. A similarity threshold is configured. If the similarity between pixels is greater than the similarity threshold, the target structure is continued to be expanded along the similarity between pixels.

[0020] If the similarity between pixels is less than or equal to the similarity threshold, the target structure expansion is stopped and the target candidate region is obtained;

[0021] Calculate the gradient strength of the target candidate area and configure the strength threshold. If the gradient strength of the target candidate area is greater than or equal to the strength threshold, the target candidate area is determined to be the target area, indicating that the image data shows that the target exists.

[0022] If the gradient intensity of the target candidate region is less than the intensity threshold, it is determined that the target candidate region is not the target region, indicating that the image data shows that the target does not exist.

[0023] As an optional implementation, if the image data shows that the target exists, then analyzing the target attributes in the image data, the target attributes including target shape, target size, target position and target density;

[0024] The target shape is quantified by the aspect ratio of the target, the target size is quantified by the sum of the number of pixels of the target, the target position is represented by the coordinates of the target, and the target density is quantified by the ratio of the sum of the number of pixels of the target to the area of ​​the target region;

[0025] The sequence attributes include the degree of target change and the rate of target reinforcement;

[0026] The target change degree is quantified by the difference between the target at adjacent time stamps, and the target enhancement rate is calculated by the signal strength of two time stamps.

[0027] As an optional implementation, the fusion logic of the target sequence attribute includes:

[0028] The target attribute and sequence attribute are normalized, and the normalized target attribute and sequence attribute are defined as vectors respectively. The weight coefficients of the target attribute and sequence attribute are configured, and the target sequence attribute is obtained by weighted fusion.

[0029] As an optional implementation, the identified comprehensive state is compared with the actual comprehensive state to obtain a feedback signal, and the feedback signal includes comprehensive state consistency and comprehensive state non-compliance. When the feedback signal is that the comprehensive state does not comply, the identification of the comprehensive state and the judgment of the target existence are optimized, and the judgment logic for optimizing the target existence includes dynamically adjusting the similarity threshold.

[0030] As an optional implementation, the function expression for dynamically adjusting the similarity threshold is as follows:

[0031] ;

[0032] In the formula, represents the dynamically adjusted similarity threshold, represents the similarity threshold of the initial configuration, represents the adjustment factor, represents the standard deviation of the target structure, Represents the standard deviation of the image data.

[0033] In a second aspect, the present application provides a recognition and evaluation method based on multimodal data analysis, the method comprising:

[0034] S1, acquiring image data and sequence data through data acquisition strategy;

[0035] S2. judging the existence of the target through the image data, and analyzing the target attributes in the image data and the sequence attributes in the sequence data. The judgment logic of the target existence includes capturing the target structure and expanding the target structure to obtain the target candidate region, and judging the existence of the target according to the gradient strength of the target candidate region;

[0036] S3, fusing the target attribute and the sequence attribute to obtain the target sequence attribute to identify the comprehensive state;

[0037] S4. Compare the identified comprehensive state with the actual comprehensive state to obtain a feedback signal, optimize the identification of the comprehensive state and the judgment of the target existence according to the feedback signal, and optimize the judgment logic of the target existence including dynamically adjusting the similarity threshold.

[0038] In a third aspect, an optional implementation of the present application further provides a computer device, a memory and a processor, wherein the memory is used to store instructions; the processor is used to execute the instructions, so that the computer device executes the above-mentioned second aspect.

[0039] In a fourth aspect, an optional implementation of the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program executes the above-mentioned second aspect when executed.

[0040] Compared with the prior art, the beneficial effects of the present application are as follows: the image data and sequence data are acquired through the data acquisition module, multimodal data is provided, the comprehensiveness and accuracy of the data are ensured, and basic data is provided for subsequent attribute analysis; the attribute analysis module determines the existence of the target through the image data, and analyzes the target attributes in the image data and the sequence attributes in the sequence data. The judgment logic of the target existence includes capturing the target structure and expanding the target structure to obtain the target candidate area, judging the existence of the target according to the gradient strength of the target candidate area, converting the original multimodal data into attributes that can be used for analysis, reducing the complexity of the multimodal data, and improving the accuracy of analysis and judgment. Attribute analysis is used to capture changes in user status; the fusion recognition module fuses the target attributes and sequence attributes to obtain the target sequence attributes to identify the comprehensive status. Through attribute fusion, multimodal data is comprehensively considered to improve the robustness and accuracy of recognition, which can more comprehensively reflect the user status; the recognition and evaluation module compares the recognized comprehensive status with the actual comprehensive status to obtain a feedback signal, and optimizes the recognition of the comprehensive status and the judgment of the target existence according to the feedback signal. The judgment logic of optimizing the target existence includes dynamically adjusting the similarity threshold and providing real-time feedback to continuously adjust and optimize the recognition method of the user status, improve the accuracy and reliability of recognition, and adapt to the dynamic changes of the user status. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0042] Figure 1 A system structure diagram of an identification and evaluation system based on multimodal data analysis provided in an embodiment of the present application;

[0043] Figure 2 A target existence judgment logic diagram of a recognition and evaluation system based on multimodal data analysis provided in an embodiment of the present application;

[0044] Figure 3 A target sequence attribute fusion logic diagram of a recognition and evaluation system based on multimodal data analysis provided in an embodiment of the present application;

[0045] Figure 4 A method flow chart of the recognition and evaluation method based on multimodal data analysis provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are clearly and completely described below in conjunction with the drawings in the specification. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0047] Example 1

[0048] like Figure 1 As shown, a system structure diagram of an identification and evaluation system based on multimodal data analysis is provided for an embodiment of the present application. The system includes a data acquisition module, an attribute analysis module, a fusion identification module and a recognition and evaluation module.

[0049] The data acquisition module is used to acquire image data and sequence data through a data acquisition strategy, and the data acquisition strategy includes controlling the consistency of the image data and the sequence data.

[0050] Data acquisition strategies include:

[0051] Adjust the equipment's parameter configuration according to the target's detection requirements, calibrate the equipment's focus method, obtain image data and sequence data through the equipment, and control the consistency of image data and sequence data.

[0052] The logic for controlling the consistency of image data and sequence data includes:

[0053] Assigning timestamps to the image data and the sequence data, matching the timestamp of the image data with the timestamp of the sequence data, and if there is a time deviation between the timestamp of the image data and the timestamp of the sequence data, determining the degree of the time deviation;

[0054] The time difference is obtained by subtracting the timestamp of the image data from the timestamp of the sequence data, and a difference threshold is configured. If the time difference is less than or equal to the difference threshold, the degree of time deviation is the degree base value, and the time deviation is corrected by interpolation calculation;

[0055] If the time difference is greater than the difference threshold, the degree of the time deviation is the extreme value, and the time deviation is corrected by weighted average calculation.

[0056] It should be understood that in order to obtain high-quality image data and sequence data, appropriate equipment should be selected, and the equipment should be calibrated and maintained regularly to ensure normal performance of the equipment; at the same time, before acquiring image data and sequence data, it is necessary to adjust the device's scanning area and resolution and other configurations according to the target detection requirements (whether it is necessary to detect the target's detailed information or the target's general framework, too high a resolution will increase the amount of data, and too low a resolution may cause information loss) to ensure that the best data quality can be obtained; when acquiring image data and sequence data, calibrate the focus mode of the equipment to ensure that the data is clear, without blur, noise or out of focus, and ensure that important details are not lost; when acquiring data multiple times, ensure the consistency of image data and sequence data for subsequent comprehensive status identification; in the process of acquiring image data and sequence data, if it is found that the image data quality or sequence data quality does not meet the requirements, it should be re-acquired in time to avoid unqualified image data and sequence data affecting subsequent analysis and identification.

[0057] It should be understood that the image data obtained here is static data, while the sequence data is dynamic data. Specifically, in the field of medical diagnosis, the image data is the imaging data of the examination site (used to determine whether there is a lesion in the examination site), while the sequence data describes whether the examination site has changed (such as dynamic information of lesion changes).

[0058] For example, in the field of medical diagnosis, image data (the patient's lung CT image data) is acquired through a device (CT machine), and corresponding sequence data (pulmonary function test sequence data) is acquired, and the parameter configuration of the CT machine (such as resolution and scanning area) is adjusted. For CT scanning, higher resolution usually requires higher radiation dose. Therefore, while ensuring the quality of image data and sequence data, the resolution is minimized to avoid unnecessary radiation; at the same time, the setting of the scanning area must cover the anatomical area required for diagnosis (such as the upstream to downstream area of ​​the lesion), and avoid redundant scanning to reduce the amount of data; for the focusing method of the CT machine, it is necessary to ensure that the X-ray source and the detector are correctly aligned to avoid image blur due to focal length error.

[0059] Specifically, the consistency problem of image data and sequence data usually comes from the difference in device sampling frequency, network delay and synchronization problems, etc. In order to control the consistency of image data and sequence data, the time deviation is corrected through the above logic, thereby reducing or eliminating the data inconsistency problem caused by the asynchronous timestamp. The interpolation calculation refers to calculating the value corresponding to the target timestamp through the previous and next timestamps, such as calculating the difference ratio between the previous and next timestamps (the amount of change per unit time) and the difference between the previous and next timestamps, and then calculating the offset of the target timestamp relative to a certain previous and next timestamps. The relationship between the timestamps can be smoothly inferred through the interpolation formula and accurately corrected. When the degree of time deviation is small, the interpolation method is used to make slight adjustments to the timestamps of the image data or sequence data to ensure that the two are more closely aligned; the weighted average calculation is to obtain the timestamp of the image data and the timestamp of the sequence data, and determine the image data and The weight of the sequence data (determined according to factors such as the frequency of data acquisition and the accuracy of timestamps. For example, image data with a lower acquisition frequency may have a higher weight because it has fewer timestamps and may be more sparse. If the timestamps of the image data are more accurate, it can also be given a higher weight; the acquisition frequency of sequence data is higher and may be denser, so its weight is usually lower, but if the timestamp accuracy of the sequence data is higher, it can also be adjusted to a higher weight), calculate the weighted sum (multiply the timestamps of the image data and sequence data by their respective weights to obtain the sum of the weighted timestamps), calculate the sum of the weights of the image data and sequence data, and the ratio of the weighted sum to the sum of the weights is the corrected timestamp to accurately correct the time deviation. When the degree of time deviation is large, the weighted average method is used to correct the timestamp, and the weighted weights are used to balance the timestamps of the image data and sequence data to obtain a reasonable correction value to avoid data synchronization distortion.

[0060] The attribute analysis module is used to determine the existence of the target through image data and analyze the target attributes in the image data. The target attributes include target shape, target size, target position and target density, and determine the sequence attributes in the sequence data. The judgment logic of the target existence includes capturing the target structure and expanding the target structure to obtain the target candidate area, and judging the target existence according to the gradient strength of the target candidate area.

[0061] The logic for judging the existence of a target is as follows: Figure 2 As shown, specifically including:

[0062] The target structure is captured by a convolutional neural network, and a target structure graph is generated. Each pixel in the target structure graph is used as a seed, and the target structure is expanded along the similarity between pixels. A similarity threshold is configured. If the similarity between pixels is greater than the similarity threshold, the target structure is continued to be expanded along the similarity between pixels.

[0063] If the similarity between pixels is less than or equal to the similarity threshold, the target structure expansion is stopped and the target candidate region is obtained;

[0064] Calculate the gradient strength of the target candidate area and configure the strength threshold. If the gradient strength of the target candidate area is greater than or equal to the strength threshold, the target candidate area is determined to be the target area, indicating that the image data shows that the target exists.

[0065] If the gradient intensity of the target candidate region is less than the intensity threshold, it is determined that the target candidate region is not the target region, indicating that the image data shows that the target does not exist.

[0066] Exemplarily, by analyzing the image data to determine whether a target exists (that is, whether a lesion exists), a convolutional neural network is first used to capture the target structure (such as the approximate structure of a lesion or the structure of a suspected lesion) from the complex image data, and finally a target structure map (lesion structure map or suspected lesion structure map) composed of multiple target structures (such as the approximate structure of a lesion or the structure of a suspected lesion) is output, thereby achieving accurate capture of the target structure (such as the approximate structure of a lesion or the structure of a suspected lesion).

[0067] Exemplarily, a pixel is selected as a seed in the target structure map (lesion structure map or suspected lesion structure map), and the similarity between adjacent pixels (grayscale difference between adjacent pixels) is analyzed based on the seed, and the target candidate area (lesion candidate area) is gradually expanded. The similarity threshold is set to ensure that only pixels with sufficient similarity to the seed pixel are included in the expansion range, which can effectively prevent misidentification. At the same time, the similarity threshold controls the accuracy of the expansion of the target candidate area (lesion candidate area). Reasonable selection of the similarity threshold can avoid over-expansion or incomplete expansion of the target candidate area (lesion candidate area). Adjusting the similarity threshold according to the texture of the target (lesion) in the image data can make the expansion of the target candidate area (lesion candidate area) more flexible and accurate, not restricted by fixed shapes, and improve the flexibility of target detection.

[0068] Exemplarily, the gradient strength of the target candidate region (lesion candidate region) is calculated (obtained by canny edge detection) to reflect the degree of edge change of the target candidate region (lesion candidate region) to further ensure the accuracy of the target region (lesion region). The gradient is often larger at the edge of the target region (lesion region). When the gradient strength of the target candidate region (lesion candidate region) is larger, it means that the target region (lesion region) has stronger edge features and meets the typical features of the target region (lesion region), and is thus determined to be the target region (lesion region). When the gradient strength of the target candidate region (lesion candidate region) is smaller, it may be noise or a non-target region (non-lesion region). Eliminating these candidate regions that do not meet the conditions to avoid misidentification of irrelevant regions can improve the robustness and reliability of target detection. This combined method can more robustly determine the existence of the target (lesion).

[0069] If the image data shows that the target exists, the target attributes in the image data are analyzed, and the target attributes include target shape, target size, target position and target density.

[0070] The target shape is quantified by the aspect ratio of the target, the target size is quantified by the sum of the number of pixels of the target, the target position is represented by the coordinates of the target, and the target density is quantified by the ratio of the sum of the number of pixels of the target to the area of ​​the target region.

[0071] Exemplarily, the target shape (lesion shape) is quantified by the aspect ratio of the target (lesion). The aspect ratio of the target (lesion) refers to the ratio of the longest axis length to the shortest axis length, which is calculated by the boundary points of the target (lesion) in the image data (the width and height of the circumscribed rectangle or bounding box of the target are extracted). When the aspect ratio is greater than 1, it usually indicates that the target (lesion) is long in shape. When the aspect ratio is close to 1, it indicates that the target (lesion) is close to a square or a circle. When the aspect ratio is less than 1, it indicates that the target (lesion) is relatively flat, and the shape of the target (lesion) can be effectively identified. The target size (lesion size) is quantified by the area of ​​the target (lesion), and the area of ​​the target (lesion) is the sum of the number of pixels representing the target (lesion) in the image data. The target (lesion) needs to be quantified. ) pixels are distinguished from background pixels, reflecting the space occupied by the target (lesion) in the image data; the target position (lesion position) is described by the coordinates of the target (lesion) in the image data, which helps to analyze the dynamic changes of the target (lesion); the target density (lesion density) is used to distinguish different types of targets (lesions), such as tumors and cysts, and is calculated by ratioing the number of pixels in the target area (lesion area) to the area of ​​the target area (lesion area). The lower the ratio, the greater the background noise, or the more gaps there are inside the target (lesion). The higher the ratio, the more concentrated the target (lesion) is in the image data and the clearer the boundary. In medical images, it may mean that the tissue in the tumor or other abnormal area is dense, which may be a manifestation of malignant tissue.

[0072] Sequence properties include the degree of target change and the rate of target reinforcement.

[0073] The degree of target change is quantified by the difference between the target at adjacent time stamps, and the target reinforcement rate is calculated by the signal strength of two time stamps.

[0074] Exemplarily, the target change degree (lesion change degree) is used to reflect the growth rate or reduction rate of the target (lesion), and is calculated by dividing the difference between the target sizes (lesion sizes) corresponding to two timestamps by the difference between the two timestamps; the target enhancement rate (lesion enhancement rate) is used to reflect the tissue characteristics of the target (lesion), by obtaining the maximum signal intensity of the target (lesion) and the timestamp corresponding to it, as well as the initial signal intensity and the timestamp corresponding to it, subtracting the maximum signal intensity of the target (lesion) from the initial signal intensity to obtain the intensity difference, subtracting the timestamp of the maximum signal intensity of the target (lesion) from the timestamp of the initial signal intensity to obtain the timestamp difference, and comparing the intensity difference with the timestamp difference to obtain the target Enhancement rate (lesion enhancement rate). If the target enhancement rate (lesion enhancement rate) is higher, it means that the target (lesion) has experienced a greater signal enhancement in a shorter period of time, which may be a manifestation of rapid development, proliferation or other abnormal phenomena of the lesion. For tumors or lesions, this may mean rapid growth or enhancement, indicating that there may be strong pathological activity. If the target enhancement rate (lesion enhancement rate) is lower, it means that the signal enhancement of the target (lesion) is slower, which may mean that the lesion grows slowly or has stabilized. If the target enhancement rate (lesion enhancement rate) is zero or negative, it means that the target (lesion) has no obvious enhancement, which may mean that the lesion is in a state of regression, degeneration or effective treatment.

[0075] The fusion recognition module is used to fuse the target attributes and sequence attributes to obtain the target sequence attributes to identify the comprehensive state.

[0076] The fusion logic of the target sequence attributes is as follows Figure 3 As shown, specifically including:

[0077] The target attribute and sequence attribute are normalized, and the normalized target attribute and sequence attribute are defined as vectors respectively. The weight coefficients of the target attribute and sequence attribute are configured, and the target sequence attribute is obtained by weighted fusion.

[0078] Exemplarily, the target attribute and the sequence attribute are compressed into the range of 0 to 1 by the minimum and maximum values ​​corresponding to the target attribute and the sequence attribute, so as to ensure that the contribution of the target attribute and the sequence attribute to the final target sequence attribute is relatively balanced, and then the normalized target attribute and the sequence attribute are defined as vectors respectively, including the target attribute vector and sequence attribute vector ,in, They represent target shape, target size, target position and target density respectively. They represent the target change degree and target reinforcement rate respectively. The final target sequence attribute vector is: ,in, and They are the weight coefficients of the target attribute and the sequence attribute, ranging from 0 to 1, and can be adjusted and These two weight coefficients control the relative importance of target attributes and sequence attributes.

[0079] Exemplarily, the target sequence attribute vector obtained by weighted fusion is input into a classification model (such as a support vector machine, a decision tree, and a neural network, etc.) for prediction, and finally a comprehensive state (that is, a user state) is obtained. In the field of medical diagnosis, the comprehensive state (user state) includes a healthy state (the user has no obvious lesions or abnormalities and is in good overall health), an early stage of lesions (the user may have mild lesions or abnormalities, the symptoms are not yet obvious, and early signs may only be slight changes in image data), a mid-stage lesion (the user's lesions or abnormalities are significantly aggravated, and the size, shape, or growth of the lesions are changing faster, which may manifest as a mid-stage or late-stage disease), a late-stage lesion (the user's condition worsens, and the image data and growth changes show accelerated lesion progression) and a stable state (the user's lesions have not changed significantly for the time being, and the size or other attributes of the lesions remain stable).

[0080] The recognition and evaluation module is used to compare the recognized comprehensive state with the actual comprehensive state to obtain a feedback signal, and optimize the recognition of the comprehensive state and the judgment of the target existence according to the feedback signal. The judgment logic of optimizing the target existence includes dynamically adjusting the similarity threshold.

[0081] The identified comprehensive state is compared with the actual comprehensive state to obtain a feedback signal, and the feedback signal includes comprehensive state compliance and comprehensive state non-compliance. When the feedback signal is that the comprehensive state is non-compliant, the identification of the comprehensive state and the judgment of the target existence are optimized. The judgment logic of optimizing the target existence includes dynamically adjusting the similarity threshold.

[0082] The function expression for dynamically adjusting the similarity threshold is as follows:

[0083] ;

[0084] In the formula, represents the dynamically adjusted similarity threshold, represents the similarity threshold of the initial configuration, represents the adjustment factor, represents the standard deviation of the target structure, Represents the standard deviation of the image data.

[0085] It should be noted that the standard deviation of the target structure It refers to the contrast of the target structure (the target structure is captured from the image data through the convolutional neural network, that is, the approximate structure of the lesion or the structure of the suspected lesion). The contrast of the target structure can be evaluated by the change of the pixel grayscale value of the target structure; the standard deviation of the image data Refers to the global contrast of the entire image data, usually the statistics of the entire image data; adjustment factor It is used to control the influence of the contrast of the target structure on the similarity threshold. In specific applications, for low contrast (weak target details and large noise), it is necessary to reduce the similarity threshold to avoid misjudgment. At this time, the adjustment factor Taking a negative value of -1 makes the adjusted similarity threshold smaller when the standard deviation of the target structure is large; for high contrast (large difference between the target and the background), the adjustment factor Taking a positive value of 1 makes the adjusted similarity threshold larger when the standard deviation of the target structure is larger, which helps to more accurately separate the target (lesion) and background in the image data.

[0086] For example, if the recognition of the comprehensive state is achieved through a support vector machine, then the optimization method for the recognition of the comprehensive state is to evaluate the effect of the comprehensive state (user state) recognition through evaluation indicators such as cross-validation, accuracy and recall rate. If the effect is not good, the accuracy of the comprehensive state (user state) recognition can be further optimized by adjusting the weight coefficients and model hyperparameters in the support vector machine.

[0087] Specifically, the similarity threshold is dynamically adjusted according to the standard deviation of the target structure and the standard deviation of the image data, so as to handle low contrast, noise and high contrast in different regions. For low contrast regions, the standard deviation of the target structure will be relatively small, so the similarity threshold is appropriately lowered to avoid misjudgment; for high contrast regions, the standard deviation of the target structure will be relatively large, so the similarity threshold will be increased, which helps to avoid low contrast regions being mistakenly considered as targets (lesions). Through the dynamic adjustment of the similarity threshold, the similarity calculation can be more intelligently processed in different target candidate regions (similarity calculation can also be obtained through distance measurement or correlation measurement, such as Euclidean distance formula, Manhattan distance formula and structural similarity, etc.), thereby improving the accuracy of target (lesion) existence judgment, and judging the target existence again according to the adjusted similarity threshold, further identifying the comprehensive state, and judging whether the identified comprehensive state is consistent with the actual comprehensive state. If not, further similarity threshold adjustment is performed until the identified comprehensive state is consistent with the actual comprehensive state.

[0088] Exemplarily, when the feedback signal is consistent with the comprehensive state, a personalized diagnosis and treatment plan is recommended based on the lesion attributes, sequence attributes and user status of the corresponding patient through the existing disease knowledge base (such as the COPD disease database) and the reasoning matching large model (clinical knowledge graph, etc.). The diagnosis and treatment plan here not only considers the patient's characteristics, but also the two-way inference of the treatment effect data corresponding to the diagnosis and treatment plans of similar historical cases.

[0089] Example 2

[0090] like Figure 4 As shown, a method flow chart of an identification and evaluation method based on multimodal data analysis is provided for an embodiment of the present application, and the method includes:

[0091] S1, acquiring image data and sequence data through data acquisition strategy;

[0092] S2. judging the existence of the target through the image data, and analyzing the target attributes in the image data and the sequence attributes in the sequence data. The judgment logic of the target existence includes capturing the target structure and expanding the target structure to obtain the target candidate region, and judging the existence of the target according to the gradient strength of the target candidate region;

[0093] S3, fusing the target attribute and the sequence attribute to obtain the target sequence attribute to identify the comprehensive state;

[0094] S4. Compare the identified comprehensive state with the actual comprehensive state to obtain a feedback signal, optimize the identification of the comprehensive state and the judgment of the target existence according to the feedback signal, and optimize the judgment logic of the target existence including dynamically adjusting the similarity threshold.

[0095] Since the principle of solving the problem by the method in the embodiment of the present application is similar to that of the system described above in the embodiment of the present application, the implementation of the method refers to the implementation of the system, and the repeated parts will not be repeated.

[0096] Example 3

[0097] In this embodiment, a computer device is provided, including a memory and a processor, wherein the memory is used to store instructions, and the processor is used to execute the instructions, so that the computer device executes the steps in the above method embodiment.

[0098] Example 4

[0099] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the steps in the above method embodiment are implemented.

[0100] The computer-readable storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and other media for storing program codes.

[0101] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the method and device described above may refer to the corresponding process in the aforementioned system embodiment, and will not be repeated here.

[0102] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0103] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application, which should be included in the scope of the claims of the present application.

Claims

1. An identification and evaluation system based on multimodal data analysis, characterized in that: include: Data acquisition module, attribute analysis module, fusion recognition module and recognition evaluation module; The data acquisition module is used to acquire image data and sequence data through a data acquisition strategy; The attribute analysis module is used to judge the existence of the target through the image data, and analyze the target attributes in the image data and the sequence attributes in the sequence data. The judgment logic of the target existence includes capturing the target structure and expanding the target structure to obtain the target candidate area, and judging the existence of the target according to the gradient strength of the target candidate area; The fusion recognition module is used to fuse the target attribute and the sequence attribute to obtain the target sequence attribute to identify the comprehensive state; The identification and evaluation module is used to compare the identified comprehensive state with the actual comprehensive state to obtain a feedback signal, and optimize the identification of the comprehensive state and the judgment of the target existence according to the feedback signal. The judgment logic of optimizing the target existence includes dynamically adjusting the similarity threshold.

2. The identification and evaluation system based on multimodal data analysis according to claim 1, characterized in that: The data acquisition strategy includes: Adjust the equipment's parameter configuration according to the target's detection requirements, calibrate the equipment's focus method, obtain image data and sequence data through the equipment, and control the consistency of image data and sequence data.

3. The identification and evaluation system based on multimodal data analysis according to claim 2, characterized in that: The logic for controlling the consistency of image data and sequence data includes: Assigning timestamps to the image data and the sequence data, matching the timestamp of the image data with the timestamp of the sequence data, and if there is a time deviation between the timestamp of the image data and the timestamp of the sequence data, determining the degree of the time deviation; The time difference is obtained by subtracting the timestamp of the image data from the timestamp of the sequence data, and a difference threshold is configured. If the time difference is less than or equal to the difference threshold, the degree of time deviation is the degree base value, and the time deviation is corrected by interpolation calculation; If the time difference is greater than the difference threshold, the degree of the time deviation is the extreme value, and the time deviation is corrected by weighted average calculation.

4. The identification and evaluation system based on multimodal data analysis according to claim 3, characterized in that: The judgment logic of the target existence includes: The target structure is captured by a convolutional neural network, and a target structure graph is generated. Each pixel in the target structure graph is used as a seed, and the target structure is expanded along the similarity between pixels. A similarity threshold is configured. If the similarity between pixels is greater than the similarity threshold, the target structure is continued to be expanded along the similarity between pixels. If the similarity between pixels is less than or equal to the similarity threshold, the target structure expansion is stopped and the target candidate region is obtained; Calculate the gradient strength of the target candidate area and configure the strength threshold. If the gradient strength of the target candidate area is greater than or equal to the strength threshold, the target candidate area is determined to be the target area, indicating that the image data shows that the target exists. If the gradient intensity of the target candidate region is less than the intensity threshold, it is determined that the target candidate region is not the target region, indicating that the image data shows that the target does not exist.

5. The identification and evaluation system based on multimodal data analysis according to claim 4, characterized in that: If the image data shows that the target exists, analyzing the target attributes in the image data, the target attributes including target shape, target size, target position and target density; The target shape is quantified by the aspect ratio of the target, the target size is quantified by the sum of the number of pixels of the target, the target position is represented by the coordinates of the target, and the target density is quantified by the ratio of the sum of the number of pixels of the target to the area of ​​the target region; The sequence attributes include the degree of target change and the rate of target reinforcement; The target change degree is quantified by the difference between the target at adjacent time stamps, and the target enhancement rate is calculated by the signal strength of two time stamps.

6. The identification and evaluation system based on multimodal data analysis according to claim 5, characterized in that: The fusion logic of the target sequence attributes includes: The target attribute and sequence attribute are normalized, and the normalized target attribute and sequence attribute are defined as vectors respectively. The weight coefficients of the target attribute and sequence attribute are configured, and the target sequence attribute is obtained by weighted fusion.

7. The identification and evaluation system based on multimodal data analysis according to claim 6, characterized in that: The identified comprehensive state is compared with the actual comprehensive state to obtain a feedback signal, and the feedback signal includes whether the comprehensive state is consistent or inconsistent. When the feedback signal shows that the comprehensive state is inconsistent, the identification of the comprehensive state and the judgment of the target existence are optimized. The judgment logic for optimizing the target existence includes dynamically adjusting the similarity threshold.

8. The identification and evaluation system based on multimodal data analysis according to claim 7, characterized in that: The function expression for dynamically adjusting the similarity threshold is as follows: ； In the formula, represents the dynamically adjusted similarity threshold, represents the similarity threshold of the initial configuration, represents the adjustment factor, represents the standard deviation of the target structure, Represents the standard deviation of the image data.

9. A recognition and evaluation method based on multimodal data analysis, implemented based on a recognition and evaluation system based on multimodal data analysis according to any one of claims 1 to 8, characterized in that: include: S1, acquiring image data and sequence data through data acquisition strategy; S2. judging the existence of the target through the image data, and analyzing the target attributes in the image data and the sequence attributes in the sequence data. The judgment logic of the target existence includes capturing the target structure and expanding the target structure to obtain the target candidate region, and judging the existence of the target according to the gradient strength of the target candidate region; S3, fusing the target attribute and the sequence attribute to obtain the target sequence attribute to identify the comprehensive state; S4. Compare the identified comprehensive state with the actual comprehensive state to obtain a feedback signal, optimize the identification of the comprehensive state and the judgment of the target existence according to the feedback signal, and optimize the judgment logic of the target existence including dynamically adjusting the similarity threshold.

10. A computer device, characterized in that: include: A memory for storing instructions; A processor is used to execute the instructions so that the computer device implements the recognition and evaluation method based on multimodal data analysis as described in claim 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the recognition and evaluation method based on multimodal data analysis as claimed in claim 9 is implemented.

Images