Self-adaptive multi-modal image recognition method and system based on artificial intelligence

Abstract

The invention relates to the field of intelligent manufacturing, in particular to a self-adaptive multi-modal image recognition method and system based on artificial intelligence, and the method comprises the steps: obtaining the image information of a target detection object in real time, and judging whether the target detection object meets a detection condition or not according to the image information; if the target detection object meets the detection condition, obtaining specification data of the target detection object; predicting the detection type of the target detection object according to the specification data; based on the detection type, identifying and generating a modal image required by the detection type, and extracting a target feature of the modal image; judging whether the target detection object meets a defect condition or not according to the target feature; and when the target detection object meets the defect condition, generating detection defect information and a modal image of the target detection object, and sending the detection defect information and the modal image to the management end. Through the self-adaptive detection scheme, different types of defects can be identified more accurately, and missing detection or false detection caused by using a unified detection mode is avoided.

Description

Technical Field

[0001] The present application relates to the field of intelligent manufacturing technology, and in particular to an adaptive multimodal image recognition method and system based on artificial intelligence. Background Art

[0002] In the intelligent manufacturing industry, fixed inspection processes and inspection locations are used. Products in each process will be sent to a fixed location for inspection when passing through the process. The inspection content is usually completed by manual or automated inspection equipment at a specific link. These inspection systems usually complete the corresponding quality inspection at the end of each process, and the position and status of the product in each link are relatively static. The inspection equipment is only responsible for quality control of specific links in the entire production process.

[0003] In the prior art, the quality inspection of each process is limited to a specific position and stage. Once a product does not go through a certain process, or there are different processing methods between processes, it is difficult for the system to effectively trace or detect the quality of the product. For complex production processes, especially production lines involving multiple product models or specifications, static inspection methods at a single process position may lead to omissions or incomplete control of the quality of multiple processes. For example, if the process of a certain link is adjusted, or a product needs to skip certain processes due to quality problems, the existing inspection system may not be able to adapt to these changes in time, resulting in missed inspections or wrong inspections. When the production line processes multiple models or specifications of products at the same time, the inspection requirements of each product may be different, and it is difficult for the existing inspection system to flexibly adapt different inspection standards between multiple processes. If the inspection of a certain link is skipped or delayed, the system is usually unable to automatically detect the problem and take remedial measures immediately, resulting in omissions in quality control.

[0004] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0005] In order to solve one or several problems in the prior art, the main purpose of this application is to provide an adaptive multimodal image recognition method and system based on artificial intelligence.

[0006] In order to achieve the above-mentioned invention object, the present application proposes an adaptive multimodal image recognition method based on artificial intelligence, the method comprising:

[0007] Acquire image information of the target object in real time, and determine whether the target object meets the detection conditions based on the image information;

[0008] If the target object meets the detection conditions, obtain the specification data of the target object;

[0009] Predicting the detection type of the target detection object according to the specification data;

[0010] Based on the detection type, identifying and generating a modality image required for the detection type, and extracting target features of the modality image;

[0011] Determining whether the target detection object meets the defect condition according to the target feature;

[0012] When the target inspection object meets the defect condition, the inspection defect information and modal image of the target inspection object are generated and sent to the management end.

[0013] The embodiment of the present application also provides an adaptive multimodal image recognition system based on artificial intelligence, comprising:

[0014] A first acquisition module is used to acquire image information of the target detection object in real time, and determine whether the target detection object meets the detection conditions according to the image information;

[0015] A second acquisition module, configured to acquire specification data of the target detection object if the target detection object meets the detection conditions;

[0016] A prediction module, used for predicting the detection type of the target detection object according to the specification data;

[0017] A recognition module, used to recognize and generate a modality image required for the detection type based on the detection type, and extract target features of the modality image;

[0018] A judgment module, used to judge whether the target detection object meets the defect condition according to the target feature;

[0019] The generation module is used to generate detection defect information and a modal image of the target detection object and send them to the management end when the target detection object meets the defect condition.

[0020] The present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0021] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0022] The adaptive multimodal image recognition method and system based on artificial intelligence in the embodiment of the present application quickly screen out targets that meet the detection conditions based on the image content, avoiding the processing of invalid data. The system intelligently predicts the detection type based on the specification data of the target detection object and selects the most suitable modal image for each detection type. For example, some objects may require high-resolution images to detect surface defects, while others may require thermal imaging or X-rays to identify internal defects. Through this adaptive detection scheme, different types of defects can be identified more accurately, avoiding missed detection or false detection caused by the use of a unified detection mode. By extracting the key features of the target and predicting based on historical data and models, it is determined whether the target meets the defect standard. Compared with traditional manual inspection methods, this intelligent defect recognition not only improves the accuracy of detection, but also maintains consistency in large-scale production and reduces errors caused by human differences. Once a defect is found, the system can immediately generate detailed defect information and corresponding modal images, and send them to the management end in real time. By intelligently predicting the detection type and optimizing the image acquisition mode, the system avoids using a unified image mode or over-calculation for all objects, significantly reducing the consumption of computing resources and hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of a flow chart of an adaptive multimodal image recognition method based on artificial intelligence according to an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a flow chart of an adaptive multimodal image recognition method based on artificial intelligence according to an embodiment of the present application;

[0025] Figure 3 This is a schematic structural block diagram of an adaptive multimodal image recognition system based on artificial intelligence according to an embodiment of the present application;

[0026] Figure 4 A schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0027] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] Reference Figure 1 In an embodiment of the present application, an adaptive multimodal image recognition method based on artificial intelligence is provided, and the method comprises:

[0030] S1. Acquire image information of the target object in real time, and determine whether the target object meets the detection conditions according to the image information;

[0031] S2. If the target object meets the detection conditions, obtain the specification data of the target object;

[0032] S3, predicting the detection type of the target detection object according to the specification data;

[0033] S4. Based on the detection type, identifying and generating a modality image required for the detection type, and extracting target features of the modality image;

[0034] S5. judging whether the target object meets the defect condition according to the target feature;

[0035] S6. When the target inspection object meets the defect condition, the inspection defect information and modal image of the target inspection object are generated and sent to the management end.

[0036] As described in the above steps S1-S3, during the production or inspection process, the image information of the target inspection object (such as parts or finished products) is captured in real time by an image acquisition device (such as a camera or sensor) installed on the production line or inspection point. This image information can be used for subsequent image analysis and processing to help determine whether the object meets the inspection conditions. By acquiring image information in real time, the system can quickly respond to changes on the production line, reduce manual intervention and delays, and improve production efficiency. This also provides the first step of image data basis for subsequent image-based defect detection and specification analysis. This step is based on a preliminary analysis of the image content to determine whether the target inspection object meets the conditions for further inspection. For example, some objects may have restrictions or requirements in terms of size, appearance, etc. These image features can be quickly analyzed by simple image processing algorithms (such as edge detection, morphological operations, etc.) and machine learning models, and preliminary filtering or classification is performed to ensure that only target objects that meet specific conditions enter the next step of the fine inspection process. This link can effectively screen out target objects that do not meet the requirements or do not need further inspection, thereby improving inspection efficiency and avoiding wasting computing resources and inspection time. This is a preprocessing step that reduces unnecessary interference for subsequent more complex judgments and analyses. Once the target object passes the initial screening, the system will further obtain the specification data related to the target. The specification data may include information such as the size, material, and production process of the object. It can be obtained through database query, docking with the production ERP system, or through a preset standardized interface. Specification data plays a key role in predicting the type of inspection and selecting the appropriate image modality. Obtaining the specification data of the target object makes subsequent inspection more intelligent and targeted. Objects of different specifications have different inspection standards and focuses in different survival programs (for example, some processing programs focus more on dimensional accuracy, while others focus on surface defects). Customized inspection by matching specification data can improve the accuracy and efficiency of inspection. Based on the acquired specification data, the system can infer what type of inspection is required for the target object. For example, if the specification of the inspection object is a high-precision part, the system may predict that its main inspection focus is dimensional accuracy; if it is a large-scale production shell, it may focus on the detection of surface defects. Based on these predictions, the system will generate appropriate modality images (such as high resolution, infrared images, etc.) for subsequent image recognition. This adaptive prediction of inspection type can greatly improve the flexibility and accuracy of inspection. By targeting different types of detection and avoiding the use of fixed standardized methods to detect all objects, unnecessary consumption of computing resources is reduced and the targeting and efficiency of detection are improved.

[0037] As described in steps S4-S6 above, according to the detection type determined in the previous step, the system selects the most suitable image acquisition mode or technology (for example, collecting details through a high-resolution camera, or using an infrared camera to capture thermal imaging information, or using X-rays to scan internal structures). Different detection types require different image modalities to obtain target features. For example, the detection of surface defects may use high-definition visual images, while the detection of internal structures may rely on CT scans or X-ray imaging. This method of selecting the appropriate image modality based on the detection type effectively avoids the "one-size-fits-all" solution, so that each type of defect can be accurately identified in the most appropriate way. It can use the optimal image mode according to the different properties of the object (such as surface, interior, size, etc.) to improve the accuracy of detection. After obtaining the appropriate modality image, the system extracts key features in the image through image processing algorithms (such as edge detection, feature point extraction, deep learning models, etc.). These features include but are not limited to shape, texture, color distribution, size, defects, etc. Deep learning technology (such as convolutional neural networks) can automatically extract more complex features from images to identify potential defects. Extracting target features is the core link of image recognition. Through in-depth analysis and learning, the most effective features related to defects can be automatically obtained from the image, which improves the intelligence and accuracy of detection. After feature extraction, the system uses pre-trained machine learning models (such as classifiers or regression models) to determine whether the target has defects. These models compare the characteristics of the target with historical data to evaluate whether the target meets the defect criteria. If the target characteristics meet the definition of defects, the system determines that the target object has defects. The automatic judgment of defects through machine learning models transforms defect detection from traditional manual judgment to an automated and accurate process. This not only improves the efficiency of defect detection, but also maintains high consistency and accuracy in large-scale production. Once a defect is detected in the target, the system generates relevant defect information (such as defect type, location, severity, etc.) and corresponding modal images (specific images or schematic diagrams of defects), and transmits this information to the management end or operator in real time. The management end can take corresponding measures based on this information (such as repair, rework or scrapping, etc.). This link ensures the rapid feedback and processing of defects and improves the controllability of product quality in the intelligent manufacturing process.

[0038] In a specific embodiment, in some production lines, although the product position of each process is sometimes fixed, the status of the product between different processes may change at any time due to the dynamic changes of the production line, the coordination between different processes and the requirements of production efficiency. For example, in the production process of assembly line, packaging line or multiple processes, the status of the product and the order of the process may not be completely fixed, and sometimes quality traceability and tracking are required. At this time, a single "static detection" may not be enough to meet the actual needs of the production line. Assume that on a complex electronic product production line, multiple processes are involved: raw material preparation, component assembly, functional testing, appearance inspection, etc. Between these processes, the product may move at different positions or stages due to factors such as production scheduling, production line adjustment, and process sequence changes. In this case, in order to ensure that the quality inspection of each process is not missed and abnormal products can be traced, it is very meaningful to detect based on the dynamic state of the product at different positions. By automatically identifying the product status at different positions through an intelligent detection system and automatically recording the detection results at the end of each process, it can be ensured that each link in the production process meets the quality requirements and avoids missed detection or mismatching. In the actual production process, the products on the production line may change according to production needs or process adjustments. If a product is skipped in a certain link or temporarily shelved due to a malfunction, the intelligent detection system can ensure that all products undergo necessary inspections to avoid omissions by monitoring the location and status of the products in real time. When a production line processes multiple products at the same time (such as mixed production of products of different models or specifications), the detection system can identify the product types in real time and adapt to different inspection standards. The inspection process for each process may be different. By dynamically inspecting the products at each position, it is ensured that each type of product is inspected according to its required standards. If a production line fails or has quality problems, it can accurately trace the inspection data of each product at different process stages, locate the root cause of the problem, and quickly troubleshoot defects in the production process.

[0039] As mentioned above, targets that meet the inspection conditions are quickly screened out based on the image content, avoiding the processing of invalid data. The system intelligently predicts the inspection type based on the specification data of the target inspection object and selects the most appropriate modal image for each inspection type. For example, some objects may require high-resolution images to detect surface flaws, while others may require thermal imaging or X-rays to identify internal defects. Through this adaptive inspection solution, different types of defects can be identified more accurately, avoiding missed detection or false detection caused by using a unified inspection mode. By extracting the key features of the target and judging whether the target meets the defect standard based on historical data and model predictions. Compared with traditional manual inspection methods, this intelligent defect recognition not only improves the accuracy of inspection, but also maintains consistency in large-scale production and reduces errors caused by human differences. Once a defect is found, the system can immediately generate detailed defect information and corresponding modal images and send them to the management end in real time. By intelligently predicting the inspection type and optimizing the image acquisition mode, the system avoids using a unified image mode or over-computing for all objects, significantly reducing the consumption of computing resources and hardware costs.

[0040] Reference Figure 2 In one embodiment, the method for extracting target features of the modality image comprises:

[0041] S41, when the detection type is the size of the target detection object, marking the target detection object in the modal image and constructing a bounding box, wherein the bounding box is used as a reference point for the size parameter;

[0042] S42, calculating a ratio parameter between a pixel size of the bounding box and an actual size according to a width and a height of the bounding box;

[0043] S43, converting the pixel size of the target detection object into an actual physical size according to the ratio parameter;

[0044] S44, extracting the size feature based on the actual physical size of the target detection object;

[0045] S45. Obtain target features based on the size features.

[0046] As described in the above steps, the target object is identified and located from the image. In this method, when the target to be detected is the size of the object, the target detection object is located and marked by an image processing algorithm (such as a convolutional neural network CNN) to mark the location of the object. A bounding box is a rectangular box used to locate objects in an image, and its boundaries are the outer edges of the object. In this step, the target detection object is framed and the spatial position of the target is defined by the bounding box. This operation provides a reference point for subsequent size calculations. By marking the target detection object and constructing a bounding box, the position of the target object in the image can be clearly obtained, thereby providing an accurate reference for subsequent size calculations. This step is the basis for size calculation and physical size conversion, ensuring that subsequent processing can focus on the target object and avoid interference from irrelevant objects. In an image, the width and height of an object are expressed in pixels, but these pixel sizes cannot directly represent the actual physical size of the object. Therefore, it is first necessary to know the pixel size (width and height) of the target object in the image, and then calculate the ratio between it and the actual physical size. The scale parameter is calculated by comparing some actual dimensions of the known object (e.g., the standard reference dimensions of the object) with the pixel dimensions measured in the image to obtain a conversion factor. This scale parameter represents the representative length of each pixel in the actual physical world. Calculating the scale parameter between the pixel dimensions and the actual dimensions ensures that the dimensional information in the image can be converted to the real physical dimensions in the real world. This ratio can be used for the dimensional measurement of any other uncalibrated object, thereby accurately estimating the size of the target object. Once the scale parameter between the pixel dimensions and the actual dimensions is calculated, the next operation is to convert the pixel dimensions (width and height) of the target object in the image by this ratio. The accurate physical dimensions can be used for tasks such as quality control and anomaly detection on the production line. Once the actual physical dimensions of the target object are obtained, the relevant dimensional features can be extracted. These features can include the aspect ratio, area, volume (for three-dimensional objects), standard deviation of the dimensions (if the fluctuation of the object dimensions needs to be considered), etc. These features help describe the distribution characteristics of the shape and size of the object. The extraction of dimensional features is crucial for subsequent defect identification. For example, if the dimensions of an object are outside the expected range, it may mean that the object has a production defect or a problem occurred during the processing. Extracting dimensional features not only helps understand the compliance of objects, but also provides useful feature inputs for machine learning models. By extracting dimensional features, detailed information can be provided for subsequent intelligent analysis to help identify objects or defects that do not meet specifications. By extracting target features, the system can fully understand the nature of the object, not only relying on dimensional features, but also combining other modal information (such as texture, color, etc.) to form a multimodal target feature set.This is very helpful for tasks such as target classification and defect detection, and can greatly improve the robustness and accuracy of detection.

[0047] In one embodiment, the method for extracting target features from the image information further includes:

[0048] When the detection type is the texture of the target detection object, marking the target detection object in the modal image;

[0049] Converting the target detection object into a grayscale image;

[0050] According to the processing results, a gray-level co-occurrence matrix is ​​constructed for the horizontally adjacent pixel pairs in the gray-level image to count the occurrence frequency of each pair of gray-level values;

[0051] Based on the constructed gray-level co-occurrence matrix, the texture characteristics of the target object are analyzed, including contrast, homogeneity and energy;

[0052] According to the analysis results, the distribution data of gray values ​​in the gray-level co-occurrence matrix is ​​further analyzed to quantify the gray distribution of the target detection object;

[0053] According to the distribution data of grayscale values, the distribution area beyond the preset grayscale range is extracted as the target feature.

[0054] As mentioned above, converting an image to a grayscale image is a step in processing the texture features of an image. A grayscale image removes color information and retains only brightness information. The grayscale value represents the brightness of each pixel in the image and is represented by a value between 0 and 255 (8-bit grayscale image). By removing color information, the complexity of the image is reduced, and texture analysis becomes more focused on the brightness changes on the surface of the object without being disturbed by color. The process of converting to a grayscale image converts the RGB value of each pixel into a single brightness value by weighted averaging. Grayscale images can more effectively reveal the texture features of the surface of an object. The conversion of grayscale images makes the subsequent texture feature calculations simpler and more focused, removes the complexity of color, and strengthens the texture information of the image. In this way, the algorithm can focus on the structure and surface features in the image without being affected by color. Gray-Level Co-occurrence Matrix (GLCM) is used to count the spatial relationship between the grayscale values ​​of pixels in an image. It captures the local texture features of an image by calculating the frequency of grayscale value pairs of pixel pairs (usually adjacent pixels) in a specific direction (such as horizontal, vertical or diagonal). In this method, after the target object is converted into a grayscale image, the algorithm constructs a grayscale co-occurrence matrix based on adjacent pixel pairs in the horizontal direction. This step provides useful data for further texture analysis by statistically analyzing the grayscale value combinations of each pair of adjacent pixels. Constructing a grayscale co-occurrence matrix can effectively capture the spatial information and structural features of the texture in the image. For example, on a two-dimensional plane, the smoothness, roughness, and contrast of the texture can be reflected by analyzing the grayscale co-occurrence matrix. This matrix is ​​the core of texture feature extraction, and it provides a key statistical basis for subsequent calculations such as contrast, homogeneity, and energy. Contrast reflects the degree of local variation of pixel grayscale in an image and indicates the roughness of the texture. A larger contrast value usually means a more complex and rough texture, and vice versa. Homogeneity describes the uniformity of the image texture. It reflects whether the distribution of grayscale values ​​is uniform. The larger the value, the smoother and more uniform the texture of the image, and the smaller the value, the greater the unevenness in the texture. Energy reflects the regularity or consistency of the texture. It is the sum of the squares of the elements in the co-occurrence matrix. The larger the value, the more regular and consistent the texture is. These texture features are usually calculated through the statistics of the grayscale co-occurrence matrix, which can effectively describe the texture, smoothness and complexity of the surface of the target object. Analyzing these texture features helps to quantitatively describe the surface characteristics of the target object. In industrial inspection, these texture features can help identify whether there are defects or irregularities on the surface of an object. For example, high contrast may indicate a rough surface, while low energy may indicate a lack of regularity on the surface. Through these features, the system can perform surface quality assessment, defect detection, etc.The distribution data of grayscale values ​​reflects the frequency distribution of each grayscale level in the image. By analyzing the distribution of grayscale values, the algorithm can identify grayscale areas or abnormal areas that appear frequently on the surface of the object. These areas may represent certain specific areas of the object, such as certain defects or special texture patterns. In this step, the histogram of grayscale values ​​or other statistical features of the grayscale co-occurrence matrix are usually analyzed to help quantify the texture distribution of the object. Analyzing the grayscale distribution helps to find possible abnormal areas on the surface of the object, especially when the distribution of certain grayscale values ​​exceeds the preset normal range, these abnormal areas may indicate defects or other quality problems. By quantifying these grayscale distributions, the system can achieve more accurate defect detection and quality assessment. Based on the grayscale value distribution analysis, a preset grayscale range is set to detect and extract those grayscale distribution areas that exceed the range. These areas that exceed the range usually represent abnormal areas in the image, such as surface defects, texture anomalies, etc. Through this operation, the algorithm can identify and extract target features that may be problem areas. The extraction of these grayscale abnormal areas helps the system to quickly locate potential defects or non-compliant parts when performing quality inspection.

[0055] In one embodiment, the method of predicting the detection type of the target detection object according to the specification data includes:

[0056] Obtaining specification data of the target detection object and image information of the current target detection object;

[0057] Based on the image information of the target object, the historical production data of the same type of object as the target object is obtained, and the change characteristics of the historical production data at different production stages are analyzed according to the time series;

[0058] Inputting the specification data and historical production data into a preset prediction model, predicting the current production stage of the target detection object through the prediction model, and outputting the prediction result;

[0059] According to the predicted production stage, the detection type of the target detection object is determined, wherein the detection type includes the size, color and texture of the target detection object.

[0060] As mentioned above, the specification data includes the physical properties of the target detection object, such as size, weight, and material. These data are obtained because they can provide important basic information for the prediction process. For example, the size may affect the appearance of the detection object and the accuracy requirements of the detection, while the material may affect the optical properties during the detection (such as reflectivity, transparency, etc.). Image information is visual feature data that can intuitively reflect the appearance of the target detection object (such as color, texture, surface defects, etc.). These image data can be used to assist in analyzing the appearance and morphology of the target detection object, thereby further affecting the prediction of the detection type. By combining specification data and image information, the physical properties and visual performance of the target detection object can be fully understood, which helps to more accurately predict its production stage and detection type. Historical production data may include performance data and production status of the target detection object at different periods and different production stages. These data can reflect the changes that may occur in the target detection object during the production process. For example, certain sizes or colors may deviate within a specific production stage, or the texture may change with different production processes. Time series analysis is a technique for analyzing the changes in data over time. By sorting the historical production data in time, the changing trends of the data in different production stages can be analyzed to find out the regular changes (such as the increase or decrease of certain features in certain stages, or the deviation pattern under certain production conditions). This helps to provide a reference for predicting the current production stage of the target detection object. Using historical production data and time series analysis can capture the common changing trends in the production process of the target detection object, and then infer the possibility of the current production stage. This method helps to identify potential problems in advance and improve the accuracy of prediction and production efficiency. The prediction model can use machine learning or deep learning algorithms to train with existing specification data and historical production data to learn the relationship between input data and the target production stage. These models usually include regression analysis, decision trees, neural networks, etc. By learning from historical data, the prediction model can predict the production stage of the target detection object based on new input data (specification data and historical data). During the training process, the model will fit the historical data by repeatedly adjusting parameters to find the most suitable prediction rules. As data continues to accumulate, the accuracy of the model prediction will also increase. By inputting specification data and historical production data into the prediction model, automated and intelligent production stage prediction can be achieved. Compared with manual judgment, the prediction model can make predictions more quickly and accurately, reduce manual intervention, and improve the efficiency of the production line. The association between production stage and detection type: Different production stages may cause different changes in the target detection object in terms of size, color, texture, etc. For example, a production stage may involve strict control of size, while another stage focuses on color uniformity or texture details. By predicting the production stage, it is possible to infer the most likely detection focus at that stage. Size, color, texture: These features are the visual and physical characteristics of the target detection object.During the production process, changes in size, color, and texture are often important bases for quality inspection. For example, certain color deviations may mean material problems, texture problems may affect the use function, and size deviations may affect the matching accuracy. Therefore, accurate definition of detection types can help factories to carry out targeted quality control. Determining the detection type according to the predicted production stage can make the detection system more targeted and ensure that the production results of each stage meet the quality standards. For example, one stage may pay more attention to dimensional accuracy, while another stage may pay more attention to appearance defects (such as uneven color or texture defects). This staged detection method helps to improve the detection efficiency of the production line and reduce the probability of false detection and missed detection.

[0061] In one embodiment, the method further comprises predicting the detection type of the target detection object according to the environmental data, the steps comprising:

[0062] Acquire the environmental data of the target detection object range in real time, and extract the target environmental features of the environmental data according to the specification data of the target detection object;

[0063] Constructing an analysis model, inputting the target environment characteristics and the image information of the target detection object into the analysis model, analyzing the influence parameters of the target environment characteristics on the target detection object through the analysis model, and outputting the analysis results;

[0064] Based on the output influencing parameters, the detection type of the target detection object is determined.

[0065] As mentioned above, environmental data includes external conditions such as temperature, humidity, air pressure, light, and air quality. Environmental factors have an important impact on the production and detection of target detection objects, especially in the production process of precision manufacturing and high-demand products. For example, temperature changes may affect the expansion or contraction of materials, humidity may affect the degree of corrosion of materials, and light conditions may affect the quality of image acquisition. By acquiring environmental data in real time, dynamic monitoring of the current production environment can be ensured. The real-time nature of this data can reflect the fluctuations in environmental conditions during the production process, thereby providing immediate data support for subsequent predictions and analysis. Real-time acquisition of environmental data enables the entire system to reflect the actual conditions of the current production environment and make timely adjustments. For example, when the temperature is too high or the humidity is too high, the production process or detection strategy may need to be adjusted immediately to avoid the impact of the environment on the target detection object. The specification data of the target detection object (such as size, material, weight, etc.) is closely related to the environmental data. Different specifications may have different sensitivities to environmental conditions. For example, some materials may be very sensitive to temperature changes, while other materials may remain stable under extreme conditions. Therefore, by extracting the target environmental features in the environmental data in combination with the specification data of the target detection object, it is possible to more accurately identify which environmental factors are most important in the production or use of the detection object. These features may include the effect of temperature on size, the effect of humidity on surface quality, the effect of light on visual inspection, etc. The extracted features will help provide suitable input for subsequent model analysis. Extracting the target environmental features can achieve the organic combination of environment and product specifications, making the analysis more specific and targeted. By effectively identifying the environmental influencing factors, unnecessary external interference can be reduced and the accuracy of model prediction can be improved. The purpose of building an analysis model is to combine the target environmental features with the image information of the target detection object for in-depth analysis. Usually, such analysis models may use methods such as machine learning, deep learning, or statistical regression. These models can learn and identify how environmental changes affect the detection results of the target detection object. For example, deep neural networks can be used to process complex image information and extract the complex relationship between environmental conditions and image features. The image information itself contains the appearance characteristics of the target detection object, such as color, texture, shape, etc., and environmental characteristics (such as light intensity, reflectivity, etc.) will directly affect the performance of these visual features. For example, strong lighting may cause overexposure of the image, while high temperature may cause changes on the surface of the material and affect the clarity of the texture. Inputting the target environment characteristics and image information into the model at the same time allows the model to fully consider the impact of environmental changes on the target object during the analysis process. This comprehensive analysis can improve the accuracy of the prediction and reduce errors caused by external environmental interference. By inputting environmental characteristics and image information into the analysis model, the system can gain a deep understanding of how environmental factors interact with the appearance characteristics of the target object.This method can identify which environmental conditions will have a significant impact on the target object, thereby achieving more accurate prediction and detection. Influence parameters are variables derived by the model based on the analysis of environmental data and image information. These parameters represent the impact of environmental factors on specific properties of the target object. For example, temperature changes may affect the size of the target object, and humidity changes may affect its surface gloss or material reflectivity. The task of the analysis model is to extract these influence parameters from the input data and evaluate their actual impact on the target object. The analysis results output by the model include the degree and direction of the impact of each environmental factor on the target object (such as the increase or decrease in size, the possibility of surface defects, etc.). These results help determine the next type of inspection and inspection standards to ensure the pertinence and efficiency of the inspection process. The output influence parameters can provide a scientific basis for determining the type of inspection. By understanding the impact of environmental factors on the target object, it is possible to accurately determine whether the inspection strategy or inspection items need to be adjusted. For example, if the temperature is too high, the model may predict that the size of the target object is biased, and the inspection process may focus on dimensional accuracy; if the humidity is too high, more attention may need to be paid to surface defects or corrosion problems. Based on the output influence parameters, the most appropriate inspection items can be determined. For example, during the production process, temperature changes have a greater impact on the size change of the target detection object, so the detection may focus on the deviation of size accuracy and shape; if the lighting conditions are poor, it may be necessary to adjust the image acquisition method or focus on the detection of texture defects. By analyzing the influencing parameters, the system can dynamically adjust the detection strategy to ensure that the target detection object can achieve the best detection effect under the current environmental conditions. Determining the detection type based on the influencing parameters can improve the accuracy and efficiency of the detection. The advantage of this is that the detection strategy is more flexible, not only taking into account the requirements of the production stage, but also adjusting in real time according to environmental conditions, minimizing the interference of the external environment on the detection results.

[0066] In one embodiment, the method of determining whether the target detection object meets the detection condition according to the image information includes:

[0067] Extracting visible features of the target detection object in the image information;

[0068] Determining whether abnormal features appear according to the visible features;

[0069] When the visible feature shows abnormal features, it is judged that the target detection object does not meet the detection conditions, and the detection defect information of the target detection object is sent to the management end based on the judgment result;

[0070] When the visible features do not show abnormal features, it is determined that the target detection object meets the detection conditions.

[0071] As mentioned above, image information contains various visual features of the target object, such as shape, color, texture, edge, size, etc. These visual features are key factors in evaluating whether the target object meets the design standards. Through computer vision technology, these features can be extracted from the image and analyzed. "Visible features" refer to physical properties related to the target object that can be directly observed in the image. For example, the surface smoothness, shape symmetry, color consistency, surface defects (such as cracks, scratches), etc. of the target object are all typical visible features. These features directly affect the quality and function of the object being tested. Generally, image processing algorithms (such as edge detection, texture analysis, color distribution analysis, etc.) are used to extract these visible features by analyzing the pixel data of the image. Convolutional neural network (CNN) in deep learning is also a commonly used image feature extraction method that can automatically learn and extract representative features from images. Extracting visible features from images is the basis for judging whether the target object meets the detection conditions. By accurately extracting these features, subsequent analysis can more effectively identify quality problems of the target object. This method can convert image information into feature data that can be understood and processed by the machine, so as to perform accurate analysis and judgment. Once the visible features of the target object are extracted, it is necessary to determine whether these features meet the preset standards or ranges. If certain features (such as shape defects, surface scratches, uneven color, etc.) deviate from the normal standards, they will be considered "abnormal features". This judgment is usually made by comparing with known standard features, or using machine learning models to identify which features do not conform to normal conditions. In practical applications, the standards for abnormal features are usually based on historical data, product specifications, or regulations in the production process. For example, for a mechanical part, there may be dimensional tolerance requirements, and exceeding the tolerance range will be considered abnormal; for products with surface coatings, scratches or bubbles are considered abnormal features. By quantifying and classifying image features, the actual extracted features are compared with predefined standards. This can be done through statistical methods, rule engines, or trained machine learning models. By making abnormal judgments on visible features, potential defects of the target object can be discovered in a timely manner. This is a key step in quality control, which ensures that only products that meet the standards can pass the inspection and prevent unqualified products from entering the market. Abnormal feature judgment not only relies on feature deviations, but also takes into account environmental and process factors. It can dynamically adapt to different production conditions and product specifications, reducing the errors and subjectivity of manual inspections. If the abnormality of the image features is confirmed, it means that the target object has defects and cannot meet the design and production requirements. Therefore, it is judged as "not meeting the test conditions". This is usually judged based on pre-defined quality standards and product requirements. Once unqualified, the system needs to promptly feedback the test results to the management end.This link is usually carried out through an automated system. The management end can be a quality monitoring system, production management system, etc., which is used to further process, record and analyze defects. Defect information may include defect type, location, severity, etc., and the system can automatically generate reports or notify relevant personnel for follow-up processing. Through automated data transmission and feedback, the management end can receive defect information in real time and take timely measures. This process reduces the time of manual intervention and improves response speed and processing efficiency. This step ensures real-time feedback and decision-making of test results. The timely transmission of defect information to the management end through automation not only improves the detection efficiency, but also ensures a rapid response to unqualified products during the production process, reducing the risks of production and quality management. This mechanism helps to discover and solve problems in advance, thereby avoiding potential quality accidents and ensuring the stability and pass rate of product quality. If there is no abnormality in the visible features extracted from the image, it can be inferred that the target detection object meets the detection standard. This means that the various features of the target detection object, such as appearance, size, shape, etc., are within an acceptable range, meet the design requirements, and meet the requirements of quality control. In actual detection, if no abnormal features are found, the system can assume that the detection item has passed the determination of the detection conditions. At this time, it can enter the production circulation or the next process stage. The judgment of this step is a "pass" signal, which means that the target object meets all necessary quality standards under the current environmental conditions. The system can automatically record this result and send a pass message to the relevant system.

[0072] In one embodiment, the target environment characteristics include: high temperature environment characteristics, high humidity environment characteristics and high vibration environment characteristics. High temperature environment: focus on surface texture. High temperature usually causes material expansion and deformation, and thermal expansion textures or cracks that are not related to the material itself may appear on the surface. If the system can sense the ambient temperature and provide real-time feedback through the temperature sensor, the detection system can use "surface texture" as the main feature, especially those thermal textures, cracks, etc. that are easily formed under temperature changes. For example, by monitoring the temperature distribution through a thermal imaging camera or an infrared sensor, the detection system can identify cracks or expansion caused by thermal stress and distinguish them from defects in the manufacturing process. High humidity environment: focus on color characteristics. In a high humidity environment, materials (especially certain plastics, wood or coatings) may absorb moisture, expand or change color, and the surface may change color, fade or other physical deformations caused by moisture. If the humidity sensor detects a higher humidity level, the system can focus on color characteristics (such as color difference, reflectivity, etc.), which may not be caused by manufacturing defects, but physical changes caused by humidity. By analyzing the color changes, it is possible to avoid treating surface color unevenness caused by humidity changes as defects. Vibration environment: Pay attention to small deformations or cracks. Vibration may cause small deformations or cracks in products, especially in dynamic environments, where microscopic defects of products may become more obvious or expand under the action of vibration. By monitoring vibration data in real time, the detection system can focus on analyzing "deformation" or "microcracks" as target features, especially in environments with high vibration intensity. These defects may appear under the influence of vibration, while conventional detection may miss these dynamic changes.

[0073] Reference Figure 3 In an embodiment of the present application, an adaptive multimodal image recognition system based on artificial intelligence is also provided, comprising:

[0074] The first acquisition module 1 is used to acquire image information of the target object in real time and determine whether the target object meets the detection conditions according to the image information;

[0075] A second acquisition module 2 is used to acquire specification data of the target detection object if the target detection object meets the detection conditions;

[0076] Prediction module 3, used for predicting the detection type of the target detection object according to the specification data;

[0077] The recognition module 4 is used to recognize and generate a modality image required for the detection type based on the detection type, and extract target features of the modality image;

[0078] A judgment module 5 is used to judge whether the target detection object meets the defect condition according to the target feature;

[0079] The generating module 6 is used to generate detection defect information and a modal image of the target detection object and send them to the management end when the target detection object meets the defect condition.

[0080] As described above, it can be understood that the various components of the adaptive multimodal image recognition system based on artificial intelligence proposed in this application can realize the functions of any one of the adaptive multimodal image recognition methods based on artificial intelligence described above, and the specific structure will not be repeated.

[0081] Reference Figure 4 In an embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as monitoring data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an adaptive multimodal image recognition method based on artificial intelligence is implemented.

[0082] The above-mentioned processor executes the above-mentioned adaptive multimodal image recognition method based on artificial intelligence, including: acquiring image information of the target detection object in real time, and judging whether the target detection object meets the detection conditions according to the image information; if the target detection object meets the detection conditions, obtaining specification data of the target detection object; predicting the detection type of the target detection object according to the specification data; based on the detection type, identifying and generating the modal image required for the detection type, and extracting the target features of the modal image; judging whether the target detection object meets the defect conditions according to the target features; when the target detection object meets the defect conditions, generating detection defect information and modal images of the target detection object and sending them to the management end.

[0083] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, an adaptive multimodal image recognition method based on artificial intelligence is implemented, comprising the steps of: acquiring image information of a target detection object in real time, and judging whether the target detection object meets detection conditions based on the image information; if the target detection object meets the detection conditions, acquiring specification data of the target detection object; predicting a detection type of the target detection object based on the specification data; based on the detection type, identifying and generating a modal image required for the detection type, and extracting target features of the modal image; judging whether the target detection object meets defect conditions based on the target features; and when the target detection object meets the defect conditions, generating detection defect information and a modal image of the target detection object and sending them to a management terminal.

[0084] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0085] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0086] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An adaptive multimodal image recognition method based on artificial intelligence, characterized in that: The method comprises: Acquire image information of the target object in real time, and determine whether the target object meets the detection conditions based on the image information; If the target object meets the detection conditions, obtain the specification data of the target object; Predicting the detection type of the target detection object according to the specification data; Based on the detection type, identifying and generating a modality image required for the detection type, and extracting target features of the modality image; Determining whether the target detection object meets the defect condition according to the target feature; When the target inspection object meets the defect condition, the inspection defect information and modal image of the target inspection object are generated and sent to the management end.

2. The adaptive multimodal image recognition method based on artificial intelligence according to claim 1, characterized in that: The method for extracting target features of the modality image comprises: When the detection type is the size of the target detection object, marking the target detection object in the modality image and constructing a bounding box, wherein the bounding box is used as a reference point for the size parameter; Calculate a ratio parameter between the pixel size of the bounding box and the actual size according to the width and height of the bounding box; According to the ratio parameter, the pixel size of the target detection object is converted into the actual physical size; Extracting the size feature based on the actual physical size of the target detection object; Based on the size feature, a target feature is obtained.

3. The adaptive multimodal image recognition method based on artificial intelligence according to claim 2, characterized in that: The method for extracting target features from the image information further comprises: When the detection type is the texture of the target detection object, marking the target detection object in the modal image; Converting the target detection object into a grayscale image; According to the processing results, a gray-level co-occurrence matrix is ​​constructed for the horizontally adjacent pixel pairs in the gray-level image to count the occurrence frequency of each pair of gray-level values; Based on the constructed gray-level co-occurrence matrix, the texture characteristics of the target object are analyzed, including contrast, homogeneity and energy; According to the analysis results, the distribution data of gray values ​​in the gray-level co-occurrence matrix is ​​further analyzed to quantify the gray distribution of the target detection object; According to the distribution data of grayscale values, the distribution area beyond the preset grayscale range is extracted as the target feature.

4. The adaptive multimodal image recognition method based on artificial intelligence according to claim 1, characterized in that: The method of predicting the detection type of the target detection object according to the specification data comprises: Obtaining specification data of the target detection object and image information of the current target detection object; Based on the image information of the target object, the historical production data of the same type of object as the target object is obtained, and the change characteristics of the historical production data at different production stages are analyzed according to the time series; Inputting the specification data and historical production data into a preset prediction model, predicting the current production stage of the target detection object through the prediction model, and outputting the prediction result; According to the predicted production stage, the detection type of the target detection object is determined, wherein the detection type includes the size, color and texture of the target detection object.

5. The adaptive multimodal image recognition method based on artificial intelligence according to claim 4, characterized in that: The method further includes predicting the detection type of the target detection object according to the environmental data, the steps comprising: Acquire the environmental data of the target detection object range in real time, and extract the target environmental features of the environmental data according to the specification data of the target detection object; Constructing an analysis model, inputting the target environment characteristics and the image information of the target detection object into the analysis model, analyzing the influence parameters of the target environment characteristics on the target detection object through the analysis model, and outputting the analysis results; Based on the output influencing parameters, the detection type of the target detection object is determined.

6. The adaptive multimodal image recognition method based on artificial intelligence according to claim 1, characterized in that: The method of judging whether the target object meets the detection condition according to the image information comprises: Extracting visible features of the target detection object in the image information; Determining whether abnormal features appear according to the visible features; When the visible feature shows abnormal features, it is judged that the target detection object does not meet the detection conditions, and the detection defect information of the target detection object is sent to the management end based on the judgment result; When the visible features do not show abnormal features, it is determined that the target detection object meets the detection conditions.

7. The adaptive multimodal image recognition method based on artificial intelligence according to claim 5, characterized in that: The target environment characteristics include: high temperature environment characteristics, high humidity environment characteristics and high vibration environment characteristics.

8. An adaptive multimodal image recognition system based on artificial intelligence, characterized in that: include: A first acquisition module is used to acquire image information of the target detection object in real time, and determine whether the target detection object meets the detection conditions according to the image information; A second acquisition module, configured to acquire specification data of the target detection object if the target detection object meets the detection conditions; A prediction module, used for predicting the detection type of the target detection object according to the specification data; A recognition module, used to recognize and generate a modality image required for the detection type based on the detection type, and extract target features of the modality image; A judgment module, used to judge whether the target detection object meets the defect condition according to the target feature; The generation module is used to generate detection defect information and a modal image of the target detection object and send them to the management end when the target detection object meets the defect condition.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images