Abnormality recognition method and device based on multiple view angles, storage medium and electronic equipment

Abstract

The invention discloses a multi-view-angle-based anomaly recognition method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring a first visual angle image of a detection object under a first visual angle and a second visual angle image of the detection object under a second visual angle; performing local feature extraction on the first view angle image and the second view angle image to obtain a plurality of first image features corresponding to the first view angle image and a plurality of second image features corresponding to the second view angle image; fusing the plurality of first image features and the first related features to obtain a first fused feature; fusing the plurality of second image features and the second related features to obtain a second fused feature; through the first fusion feature and the second fusion feature, an anomaly recognition result of the detection object is obtained, and the method can be applied to artificial intelligence scenes and relates to technologies such as computer vision. According to the method and the device, the technical problem of relatively low anomaly recognition accuracy based on multiple view angles is solved.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method, apparatus, storage medium, and electronic device for anomaly identification based on multiple perspectives. Background Technology

[0002] In multi-view anomaly recognition scenarios, image information is usually simply captured from multiple perspectives without interaction or fusion of these different perspectives, thus failing to truly grasp and utilize the advantages brought by multiple perspectives.

[0003] Without interaction or fusion, image information from each viewpoint is isolated, requiring individual processing of each viewpoint without capturing the correlation and complementarity between different viewpoints. This leads to low accuracy in anomaly detection based on multiple viewpoints. Therefore, there is a problem with low accuracy in anomaly detection based on multiple viewpoints.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and electronic device for anomaly identification based on multiple perspectives, in order to at least solve the technical problem of low accuracy in anomaly identification based on multiple perspectives.

[0006] According to one aspect of the embodiments of this application, a multi-view anomaly recognition method is provided, comprising: acquiring a first-view image of a detected object from a first viewpoint and a second-view image of the detected object from a second viewpoint; performing local feature extraction on the first-view image and the second-view image to obtain a plurality of first image features corresponding to the first-view image and a plurality of second image features corresponding to the second-view image; fusing the plurality of first image features and a first related feature to obtain a first fused feature, wherein the first related feature is a related feature among the plurality of second image features whose correlation with the plurality of first image features is greater than or equal to a first threshold; fusing the plurality of second image features and the second related feature to obtain a second fused feature, wherein the second related feature is a related feature among the plurality of first image features whose correlation with the plurality of second image features is greater than or equal to a second threshold; and obtaining an anomaly recognition result of the detected object through the first fused feature and the second fused feature.

[0007] According to another aspect of the embodiments of this application, a multi-view anomaly recognition device is also provided, comprising: an acquisition unit, configured to acquire a first-view image of a detected object from a first viewpoint and a second-view image of the detected object from a second viewpoint; an extraction unit, configured to perform local feature extraction on the first-view image and the second-view image to obtain a plurality of first image features corresponding to the first-view image and a plurality of second image features corresponding to the second-view image; a first fusion unit, configured to fuse the plurality of first image features and a first related feature to obtain a first fusion feature, wherein the first related feature is a related feature among the plurality of second image features whose correlation with the plurality of first image features is greater than or equal to a first threshold; a second fusion unit, configured to fuse the plurality of second image features and the second related feature to obtain a second fusion feature, wherein the second related feature is a related feature among the plurality of first image features whose correlation with the plurality of second image features is greater than or equal to a second threshold; and a recognition unit, configured to obtain anomaly recognition results of the detected object through the first fusion feature and the second fusion feature.

[0008] As an optional approach, the plurality of first image features include first region features and second region features, and the processing of the plurality of second image features includes M region features, where M is an integer greater than 1. The first fusion unit includes: a first fusion module, used to fuse the first related region feature and the first region feature to obtain a new first region feature, wherein the first related region feature is a region feature among the M region features whose correlation with the first region feature is greater than or equal to a first threshold; a second fusion module, used to fuse the second related region feature and the second region feature to obtain a new second region feature, wherein the second related region feature is a region feature among the M region features whose correlation with the second region feature is greater than or equal to the first threshold; and a combination module, used to combine the new first region feature and the new second region feature to obtain the first fused feature.

[0009] As an optional solution, the above-mentioned device further includes: a first correlation module, used to calculate a correlation matrix between the first regional feature and each of the M regional features before fusing the first correlated regional feature and the first regional feature to obtain a new first regional feature, wherein the correlation matrix is ​​used to represent the semantic correlation degree between the first regional feature and each of the M regional features; and a second correlation module, used to determine, based on the correlation matrix, the top N regional features in terms of semantic correlation degree from the M regional features before fusing the first correlated regional feature and the first regional feature to obtain a new first regional feature, wherein N is an integer greater than 0.

[0010] As an optional solution, the above apparatus further includes: a first mapping module, configured to perform linear mapping on the first region features before fusing the first relevant region features and the first region features to obtain a new first region feature, converting the first region features into query vectors; a second mapping module, configured to perform linear mapping on the M region features respectively before fusing the first relevant region features and the first region features to obtain M key vectors and M value vectors; the apparatus further includes: a first vector module, configured to determine, based on the relevance matrix, the top N region features ranked by semantic relevance from the M region features. Then, from the M key vectors, N key vectors corresponding to the N regional features are determined; the second vector module is used to determine N value vectors corresponding to the N regional features from the M value vectors after determining the top N regional features in semantic relevance ranking from the M regional features according to the relevance matrix; the first fusion module includes: a first fusion submodule, used to integrate the query vector, the N key vectors, and the N value vectors using a cross-attention mechanism to obtain an enhanced first regional feature; and a second fusion submodule, used to adjust the enhanced first regional feature to the original shape of the first regional feature to obtain the new first regional feature.

[0011] As an optional solution, the extraction unit includes: a first model module, used to input the first viewpoint image and the second viewpoint image into a first image processing model to obtain the plurality of first image features and the plurality of second image features output by the first image processing model, wherein the first image processing model is used to extract the original features of the image; the recognition unit includes: a second model module, used to input the first fusion feature and the second fusion feature into a second image processing model to obtain the first target feature and the second target feature output by the second image processing model, wherein the second image processing model is used to process abnormal image features into positive ones. The system comprises: a common image feature module; a first acquisition module, configured to acquire the plurality of first image features and compare them with the first target feature to obtain a first comparison result, wherein the first comparison result is used to indicate the feature difference between the plurality of first image features and the first target feature; a second acquisition module, configured to acquire the plurality of second image features and compare them with the second target feature to obtain a second comparison result, wherein the second comparison result is used to indicate the feature difference between the plurality of second image features and the second target feature; and a third acquisition module, configured to acquire the anomaly recognition result based on the first comparison result and the second comparison result.

[0012] As an optional solution, the above-mentioned device further includes: a fourth acquisition module, used to acquire a first image processing model as a teacher model before inputting the first viewpoint image and the second viewpoint image into the first image processing model to obtain the plurality of first image features and the plurality of second image features output by the first image processing model; a fifth acquisition module, used to acquire a second image processing model to be trained as a student model before inputting the first viewpoint image and the second viewpoint image into the first image processing model to obtain the plurality of first image features and the plurality of second image features output by the first image processing model; and a training module, used to train the student model using the teacher model through the teacher model to obtain the second image processing model before inputting the first viewpoint image and the second viewpoint image into the first image processing model to obtain the plurality of first image features and the plurality of second image features output by the first image processing model.

[0013] As an optional approach, the training module includes: an execution submodule, configured to perform the following steps until the trained second image processing model is obtained: determining the current normal image from the plurality of normal images and using the current normal image as the current training sample; inputting the current training sample into the trained first image processing model, and having the trained first image processing model perform multi-scale feature encoding on the current training sample to obtain K first initial features output by the trained first image processing model at K scales; performing local feature extraction on the K first initial features respectively to obtain multi-view image features at each of the K scales; fusing the multi-view image features at each scale to obtain multi-view fused features at each of the K scales; and integrating the multi-view fused features at each scale to obtain multi-scale integrated features. Given a current second image processing model to be trained, the multi-scale integrated features are input into the current second image processing model, and the current second image processing model performs multi-scale feature decoding on the multi-scale integrated features to obtain K second initial features output by the current second image processing model at the K scales. The mean square error between the initial features at the same scale among the K first initial features and the K second initial features is obtained to obtain K mean square errors. The K mean square errors are summed to obtain a loss value. If the loss value does not meet the convergence condition, the next normal image is determined from the multiple normal images and used as the current training sample. If the loss value meets the convergence condition, the current second image processing model is determined as the trained second image processing model.

[0014] As an optional solution, the aforementioned identification unit includes: a sixth acquisition module, configured to acquire a first anomaly score of the detected object from a first perspective using the first fusion feature, wherein the first anomaly score indicates the probability or degree of anomaly of the detected object from the first perspective; a seventh acquisition module, configured to acquire a second anomaly score of the detected object from a second perspective using the second fusion feature, wherein the second anomaly score indicates the probability or degree of anomaly of the detected object from the second perspective; and a ninth acquisition module, configured to acquire the anomaly identification result based on the first anomaly score and the second anomaly score.

[0015] As an optional solution, the sixth acquisition module includes: a first acquisition submodule, used to acquire pixel-level anomaly scores corresponding to each pixel position on the first viewpoint image through the first fusion feature, wherein the pixel-level anomaly scores are used to indicate the probability or degree of anomaly at the pixel position; and a second acquisition submodule, used to determine the pixel-level anomaly score that indicates the highest probability or degree of anomaly at the pixel position as the first anomaly score.

[0016] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the multi-view-based anomaly identification method as described above.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described multi-view anomaly recognition method through the computer program.

[0018] In this embodiment, images of the detected object are first captured from two different perspectives: a first-view image from a first perspective and a second-view image from a second perspective. This ensures the diversity of information and provides a rich data source for subsequent feature extraction and fusion.

[0019] Next, feature extraction is performed on these two images, yielding multiple first image features and multiple second image features. This helps capture key information in the images, providing strong support for subsequent anomaly detection.

[0020] Next, feature fusion is performed. This is not a simple feature overlay, but a targeted fusion based on the correlation between features. Specifically, multiple first image features are fused with highly correlated portions (i.e., first correlated features) from multiple second image features to obtain the first fused feature. Similarly, multiple second image features are fused with highly correlated portions (i.e., second correlated features) from multiple first image features to obtain the second fused feature. This fusion method preserves the unique information from each viewpoint while capturing the correlation and complementarity between different viewpoints.

[0021] Finally, by using these two sets of fused features, the anomaly identification results of the detected object can be obtained more comprehensively and accurately. This is because these two sets of features not only contain unique information from each perspective but also integrate relevant information from different perspectives, thereby achieving the goal of more comprehensively and accurately identifying anomalies in the detected object. This improves the technical accuracy of anomaly identification based on multiple perspectives and solves the technical problem of low accuracy in anomaly identification based on multiple perspectives. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a schematic diagram of an application environment for an optional multi-view anomaly recognition method according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the flow of an optional multi-view anomaly recognition method according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an optional multi-view anomaly recognition method according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of another optional multi-view anomaly recognition method according to an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of an optional multi-view anomaly recognition device according to an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] For ease of understanding, the following terms are explained:

[0032] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0033] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0034] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, detecting, and measuring targets, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream tasks after fine-tuning. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0035] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and pre-trained learning. Pre-trained models are the latest development in deep learning, integrating all of these techniques.

[0036] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0037] The solutions provided in this application involve artificial intelligence technologies such as computer vision and machine learning, which are specifically illustrated through the following embodiments:

[0038] According to one aspect of the embodiments of this application, a multi-view anomaly identification method is provided. Optionally, as an optional implementation, the above-described multi-view anomaly identification method can be applied to, but is not limited to, [examples of other methods]. Figure 1 The environment shown may include, but is not limited to, user equipment 102 and server 112. User equipment 102 may include, but is not limited to, a display 104, a processor 106 and a memory 108. Server 112 includes a database 114 and a processing engine 116.

[0039] The specific process can be summarized in the following steps:

[0040] In step S102, the user equipment 102 acquires a first-view image of the detected object from a first-view perspective and a second-view image of the detected object from a second-view perspective.

[0041] Step S104: Send the first-view image and the second-view image to the server 112 via network 110;

[0042] In steps S106-S110, server 112 performs local feature extraction on the first-view image and the second-view image through processing engine 116 to obtain (multiple) first image features corresponding to the first-view image and (multiple) second image features corresponding to the second-view image. It further fuses the multiple first image features and first related features to obtain a first fused feature, and fuses the multiple second image features and second related features to obtain a second fused feature. Finally, it obtains the anomaly recognition result of the detected object through the first fused feature and the second fused feature.

[0043] In step S112, the anomaly identification result is sent to the user equipment 102 via the network 110. The user equipment 102 displays the anomaly identification result on the display 104 via the processor 106 and stores the anomaly identification result in the memory 108.

[0044] remove Figure 1Beyond the examples shown, the terminal devices described above can be terminal devices configured with a target client, including but not limited to at least one of the following: mobile phones (such as Android phones, iOS phones, etc.), laptops, tablets, PDAs, MIDs (Mobile Internet Devices), PADs, desktop computers, smart TVs, etc. The target client can be a video client, instant messaging client, browser client, educational client, etc. The networks described above can include, but are not limited to, wired networks and wireless networks. The wired networks include local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). The wireless networks include Bluetooth, Wi-Fi, and other networks that enable wireless communication. The server described above can be a single server, a server cluster consisting of multiple servers, or a cloud server. The above is merely an example, and no limitations are imposed in this embodiment.

[0045] Alternatively, as an alternative implementation method, such as Figure 2 As shown, the multi-view anomaly detection method can be executed by an electronic device, such as... Figure 1 The user equipment or server shown includes the following specific steps:

[0046] S202, acquire the first-view image of the detected object from the first viewpoint and the second-view image of the detected object from the second viewpoint;

[0047] S204, Local feature extraction is performed on the first viewpoint image and the second viewpoint image to obtain multiple first image features corresponding to the first viewpoint image and multiple second image features corresponding to the second viewpoint image.

[0048] S206-1, multiple first image features and first related features are fused to obtain a first fused feature, wherein the first related feature is a related feature among multiple second image features whose correlation with multiple first image features is greater than or equal to a first threshold;

[0049] S206-2, multiple second image features and second related features are fused to obtain a second fused feature, wherein the second related feature is a related feature among multiple first image features whose correlation with multiple second image features is greater than or equal to a second threshold;

[0050] S208, obtain the anomaly identification result of the detected object through the first fusion feature and the second fusion feature.

[0051] Optionally, in this embodiment, the above-described multi-view anomaly recognition method can be applied, but is not limited to, artificial intelligence scenarios, such as in intelligent manufacturing production lines where product quality inspection is a crucial step. By installing cameras at different angles, images of the product from different perspectives can be acquired.

[0052] Specifically, two (or more) cameras are installed on the production line to capture images of the product from different angles. For example, one camera might capture the product from the front, while another might capture it from the side or top.

[0053] Further utilize deep learning algorithms, such as convolutional neural networks (CNNs), to extract features from the image at each viewpoint. These features may include key information such as shape, color, and texture. Analyze the correlation between the features extracted from the two viewpoints. By setting a correlation threshold, determine which features are highly correlated. Fuse the features from each viewpoint with highly correlated features from the other viewpoint. This fusion can be achieved through weighted averaging, feature concatenation, or other complex fusion techniques. Based on the fused features, train an anomaly detection model. These models can use algorithms such as support vector machines (SVMs), random forests, or neural networks.

[0054] As new products pass through the production line, multi-view images are captured again, features are extracted and fused, and then a trained model is used for anomaly identification. If an anomaly is detected, the (smart manufacturing production line) system can immediately trigger an alarm to notify the operator to intervene. In severe cases, the system can even automatically stop the production line to prevent defective products from entering the next production stage.

[0055] Optionally, in this embodiment, obtaining a first-view image of the detected object from a first-view perspective and a second-view image from a second-view perspective can refer to using a photographic device (such as a camera or video camera) to capture images of the same detected object from two different spatial angles, thereby obtaining two images from different perspectives.

[0056] However, acquiring monetary images from multiple perspectives is not limited to just two perspectives. The first and second examples are merely illustrative; even more perspectives can be added. For instance, in fields such as 3D modeling, virtual reality, and augmented reality, images may be captured from dozens or even hundreds of different angles to more accurately reconstruct or display an object.

[0057] To illustrate further, consider the optional assumption that you want to capture comprehensive images of a three-dimensional object, such as a sculpture. You could take a photograph from the front of the sculpture to obtain a "first-view image"; then take another photograph from the side or another angle to obtain a "second-view image".

[0058] Optionally, feature extraction can refer to using algorithms to extract key information from an image, such as edges, corners, and textures; this information is called image features. In this embodiment, "performing local feature extraction on the first-view image and the second-view image" can mean extracting their respective features from the first-view image and the second-view image, or identifying and extracting key information or features of local regions from the image. These features may include information such as color, texture, shape, and edges, i.e., "multiple first image features" and "multiple second image features".

[0059] To further illustrate, the image (first-view image / second-view image) can optionally be segmented into different regions. These regions can be segmented based on image attributes such as color, texture, and shape. For example, an image segmentation algorithm can be used to divide the image into several homogeneous regions. This embodiment does not limit the region segmentation method; superpixel segmentation techniques, such as the SLIC (Simple Linear Iterative Clustering) algorithm, can also be used. This method divides the image into a series of compact, uniform superpixel regions based on the similarity and spatial proximity between pixels.

[0060] After region segmentation, the next step is to extract features from these regions. The purpose of feature extraction is to capture key information in the image to facilitate subsequent operations such as classification, recognition, or matching. For example, features such as color histograms and color moments can be extracted for each region to describe its color distribution. Alternatively, gray-level co-occurrence matrices and filters can be used to extract texture information, which reflects the spatial relationships and arrangement patterns between pixels in the image. For regions with obvious shape features, boundary descriptors (such as Fourier transforms and Hu moments) can be extracted to characterize shape information.

[0061] By extracting local features from first-view and second-view images, two sets of feature vectors describing the image content can be obtained. These feature vectors capture key information in the image, providing strong data support for subsequent tasks such as classification, recognition, and comparison. Especially in applications such as multi-view anomaly detection, features extracted from different perspectives can complement each other, thereby improving the accuracy and robustness of recognition.

[0062] Optionally, in this embodiment, feature fusion can refer to combining features from different sources or different perspectives in a certain way to generate a new, more expressive, and robust feature set. For example, "multiple first image features" and "multiple second image features" are features extracted from first-view images and second-view images, respectively. "First related features" refer to features among multiple second image features that are highly correlated with the multiple first image features, and "second related features" refer to features among multiple first image features that are highly correlated with the multiple second image features. The fusion process is based on the correlation between these features.

[0063] Optionally, in this embodiment, the correlation between features can be measured in various ways, such as cosine similarity, Pearson correlation coefficient, etc. Choosing an appropriate threshold to determine which features are "highly correlated" is an important step and may need to be adjusted based on the specific task and dataset. Feature fusion methods include weighted averaging, concatenation, or using more complex fusion strategies.

[0064] To further illustrate, consider the optional assumption that a scene was captured from two different angles, and features were extracted from these two images. The features from these two perspectives are then fused. Specifically: First, the correlation between multiple first image features and multiple second image features is calculated. If a certain second image feature is highly correlated with a portion of a set first image feature (exceeding a predetermined first threshold), then this second image feature is considered a "first correlated feature" and is fused with the corresponding first image feature. Similarly, multiple first image features highly correlated with multiple second image features (i.e., "second correlated features") are also identified and fused.

[0065] By fusing features from another perspective that are highly correlated with those from each perspective, a richer and more robust feature set can be obtained. This fusion method leverages complementary information between different perspectives, which helps improve the performance of subsequent tasks, such as anomaly detection and classification. Especially when dealing with complex scenes or objects, multi-view feature fusion can provide more contextual information and details, thereby enhancing the model's judgment and accuracy.

[0066] Optionally, in this embodiment, the anomaly identification result can be the result of analyzing the first fusion feature and the second fusion feature, using a certain algorithm or model to determine whether the detected object is abnormal or deviates from the normal state. These fusion features contain key information captured from different perspectives and filtered and fused, providing a rich data foundation for anomaly identification.

[0067] To further illustrate, an optional assumption is that this method is used in quality control within the manufacturing industry. A product is photographed from two different angles, and corresponding features are extracted. These features are then fused to form a first fused feature and a second fused feature. Now, a trained anomaly detection model is used to analyze these fused features. If the model detects that certain features match known anomaly patterns or deviate from the normal feature distribution, it outputs an anomaly identification result, indicating that the product may have a problem.

[0068] It should be noted that this embodiment significantly improves the accuracy of anomaly identification by combining image features from different perspectives and fusing the correlations between features. It not only captures information that might be missed from a single perspective but also enhances the expression of useful information through feature fusion, thereby more accurately identifying anomalies or defects in the detected object.

[0069] To further illustrate, optional examples include... Figure 3 As shown, a first-view image 302-1 of the detected object captured by image acquisition device A from a first-view perspective, and a second-view image 302-2 of the detected object captured by image acquisition device B from a second-view perspective are acquired. Feature extraction is performed on the first-view image 302-1 and the second-view image 302-2 to obtain multiple first image features 304-1 corresponding to the first-view image 302-1 and multiple second image features 304-2 corresponding to the second-view image 302-2. The multiple first image features 304-1 and the first related features are fused to obtain the first fused feature 306-1. The first relevant feature is one of the multiple second image features 304-2 that has a correlation with the multiple first image features 304-1 that is greater than or equal to a first threshold. The multiple second image features 304-2 and the second relevant feature are fused to obtain a second fused feature 306-2. The second relevant feature is one of the multiple first image features 304-1 that has a correlation with the multiple second image features 304-2 that is greater than or equal to a second threshold. The anomaly identification result 308 of the detected object is obtained through the first fused feature 306-1 and the second fused feature 306-2.

[0070] The embodiments provided in this application first capture images of the detected object from two different perspectives: a first-view image from a first perspective and a second-view image from a second perspective. This ensures the diversity of information and provides a rich data source for subsequent feature extraction and fusion. Next, feature extraction is performed on these two images, yielding multiple first-image features and multiple second-image features. This helps capture key information in the images, providing strong support for subsequent anomaly identification. Then, feature fusion is performed. This is not a simple feature superposition, but a targeted fusion based on the correlation between features. Specifically, multiple first-image features are fused with highly correlated portions (i.e., first correlated features) of multiple second-image features to obtain a first fused feature. Similarly, multiple second-image features are fused with highly correlated portions (i.e., second correlated features) of multiple first-image features to obtain a second fused feature. This fusion method preserves the unique information from each perspective while capturing the correlation and complementarity between different perspectives. Finally, through these two sets of fused features, the anomaly identification results of the detected object can be obtained more comprehensively and accurately. Because these two sets of features not only contain unique information from each perspective, but also integrate relevant information from different perspectives, they achieve the goal of more comprehensively and accurately identifying anomalies in the detected object, thereby improving the technical effect of anomaly identification based on multiple perspectives.

[0071] As an optional approach, multiple first image features and first related features are fused to obtain a first fused feature, including:

[0072] S1-1, process multiple first image features into first region features and second region features, and process multiple second image features into M region features, where M is an integer greater than 1;

[0073] S1-2, the first relevant region feature and the first region feature are fused to obtain a new first region feature, wherein the first relevant region feature is a region feature among M region features whose correlation with the first region feature is greater than or equal to the first threshold.

[0074] S1-3, the second relevant region feature and the second region feature are fused to obtain a new second region feature, wherein the second relevant region feature is a region feature among M region features whose correlation with the second region feature is greater than or equal to the first threshold.

[0075] S1-4, combine the new first region feature and the new second region feature to obtain the first fused feature.

[0076] Optionally, in this embodiment, the image features are first processed into different regional features; then, based on the correlation between the regional features, they are selectively fused with relevant regional features in another image; finally, these fused regional features are combined to form the final fused features.

[0077] In specific processing methods, features can be divided into different regions based on the spatial location of the image. For example, for a face image, features of the eyes, nose, and mouth can be divided into the first region, while features of the cheeks, forehead, and chin can be divided into the second region.

[0078] Optionally, features can be categorized based on their type or nature. For example, some features may emphasize texture information, while others may emphasize shape or color information. These different types of features can be categorized into first-region features and second-region features, respectively.

[0079] Optionally, clustering algorithms (such as K-means, spectral clustering, etc.) can be used to perform cluster analysis on multiple first image features. By setting the number of clusters to 2, the features can be divided into two clusters, corresponding to the first region features and the second region features respectively. In this way, the similarity between features can be automatically discovered and classified.

[0080] It should be noted that this embodiment describes the specific steps of fusing multiple first image features with a first related feature to obtain a first fused feature. This process includes feature processing, correlation judgment, region feature fusion, and final combination of the fused features. The same method can also be applied to the fusion of multiple second image features and second related features. Furthermore, this embodiment can also be applied to fusing multiple second image features and second related features to obtain a second fused feature; the first and second are merely illustrative examples.

[0081] To further illustrate, consider two images of the same scene taken from different angles. The features of each image can be processed into multiple region features, such as an upper region and a lower region. Then, the correlation between the upper region features in the first image and the various region features in the second image is analyzed. The region features in the second image most relevant to the upper region of the first image are found and fused. Similarly, this process is performed on other regions of the first image. Finally, all the fused region features are combined to form the first fused feature. The same process applies to generating the second fused feature.

[0082] Furthermore, in this embodiment, processing multiple first image features into first region features and second region features is merely illustrative; multiple first image features can be processed into a greater number of region features. For example, consider two images of the same building taken from different angles, image A and image B. To simplify the explanation, each image is divided into four regions: top left, top right, bottom left, and bottom right. Therefore, each image will have four region features.

[0083] For image A, four region features are extracted: A_top left, A_top right, A_bottom left, and A_bottom right. For image B, four region features are also extracted: B_top left, B_top right, B_bottom left, and B_bottom right.

[0084] For the top-left region feature A, analyze its correlation with all region features in image B. Assuming that after analysis, the top-left region A has the highest correlation with the top-left region B (greater than or equal to the first threshold), then merge the top-left region A and the top-left region B to obtain a new top-left region feature A.

[0085] Similarly, for A_top right, assuming it has the highest correlation with B_top right (greater than or equal to the first threshold), A_top right and B_top right are fused to obtain a new A_top right region feature.

[0086] The same operation is performed on A_bottom left and A_bottom right, respectively, and they are fused with the region features with the highest correlation in image B.

[0087] Finally, the four new regions—top left, top right, bottom left, and bottom right—are combined to form the first fused feature. This first fused feature integrates relevant information from various regions in both image A and image B. The same process can be applied to image B to generate the second fused feature.

[0088] The embodiments provided in this application enable more refined utilization of complementary information between different perspectives. This fusion method considers not only the global correlation between features but also the correlation between local regions, thereby more accurately capturing and representing key information in images. Therefore, this method significantly improves the performance of tasks such as image recognition, classification, and reconstruction. Especially when dealing with complex scenes or objects, this refined feature fusion method can provide more detail and contextual information.

[0089] As an optional approach, before fusing the first relevant region features and the first region features to obtain a new first region feature, the method further includes:

[0090] S2-1, Calculate the correlation matrix between the first regional feature and each regional feature among the M regional features, where the correlation matrix is ​​used to represent the semantic correlation between the first regional feature and each regional feature among the M regional features;

[0091] S2-2, Based on the correlation matrix, determine the top N region features with the highest semantic relevance from the M region features, where N is an integer greater than 0.

[0092] Optionally, in this embodiment, the correlation matrix can be understood as a mathematical matrix used to represent the semantic correlation between two different regional features, such as representing the semantic correlation between a first regional feature and M regional features in a second-view image. The semantic correlation can be understood as the semantic similarity or association between two regional features; a high semantic correlation can mean that the two regional features have a high degree of similarity in content.

[0093] It should be noted that before fusion, it is necessary to determine which regional features are highly correlated. To this end, this embodiment first calculates the correlation matrix between the first regional features and all regional features in the second-view image. This matrix can quantitatively represent the semantic correlation between the two. Subsequently, based on this correlation matrix, the N regional features most correlated with the first regional features are selected, ensuring that the correlation between these selected regional features and the first regional features meets a preset first correlation condition, that is, ensuring that these selected regional features are considered the first correlated regional features.

[0094] To further illustrate, consider two images: Image A (first-person perspective) shows a front view of a car, and Image B (second-person perspective) shows a side view of the car. This embodiment aims to fuse features about the car from these two perspectives. First, a first region feature about the car's front is extracted from Image A. Then, the correlation matrix between this region feature and all region features in Image B is calculated. This matrix reveals that a certain region feature about the car's side in Image B is highly correlated with the front feature in Image A. Based on this correlation ranking, the N most relevant region features are selected and fused with the first region feature to obtain a new region feature that integrates information from both perspectives.

[0095] The embodiments provided in this application introduce a correlation matrix and a ranking mechanism, which enables more accurate identification of regional features in the second-view image that are most relevant to the features of the first region. This precise matching and fusion helps improve the accuracy of subsequent anomaly identification.

[0096] As an alternative solution,

[0097] Before fusing the first relevant region features and the first region features to obtain new first region features, the method also includes:

[0098] S3-1, perform linear mapping on the features of the first region and convert the features of the first region into a query vector;

[0099] S3-2, perform linear mapping on the M regional features respectively to obtain M key vectors and M value vectors;

[0100] After determining the top N semantically relevant regional features from M regional features based on the relevance matrix, the method further:

[0101] S4-1, From M key vectors, determine N key vectors corresponding to N regional features;

[0102] S4-2, from M value vectors, determine N value vectors corresponding to N regional features;

[0103] The first relevant region feature and the first region feature are fused to obtain a new first region feature, including: using a cross attention mechanism to integrate the query vector, N key vectors, and N value vectors to obtain an enhanced first region feature; and adjusting the enhanced first region feature to the original shape of the first region feature to obtain a new first region feature.

[0104] Optionally, in this embodiment, linear mapping may refer to transforming the original regional features into another representation through some linear transformation (usually through a linear matrix), making the features more suitable for subsequent calculations or comparisons.

[0105] Furthermore, linear mapping can be viewed as a mathematical transformation of the original features, which does not change the relative relationships between features, but may change the dimension or representation of the features. The query vector is used to compare with other key vectors to determine the relevance between them.

[0106] Optionally, in this embodiment, the query vector can be a vector obtained by transforming the features of the first region through a linear mapping. In the attention mechanism, the query vector represents the currently interested information or the focus of the "query".

[0107] Optionally, in this embodiment, the key vector can be a vector obtained by transforming each of the M region features through a linear mapping. In the attention mechanism, the key vector is used to compare with the query vector to calculate the attention weight. Each key vector represents a potential point of interest, and its similarity to the query vector determines the degree of attention it receives.

[0108] Optionally, in this embodiment, the value vector can also be derived from the M region features through a linear mapping, corresponding one-to-one with the key vector. In the attention mechanism, once the key vector most relevant to the query vector is determined, the corresponding value vector is used to calculate the output result. The value vector contains the actual information to be focused on, and its weight is determined by the relevance between the query vector and the key vector.

[0109] Optionally, in this embodiment, the cross-attention mechanism can be understood as an attention mechanism that allows the model to simultaneously pay attention to information from different sources and perform weighted integration based on their correlation.

[0110] To further illustrate, let's take the query vector from the current perspective as Q, and determine M key vectors and M value vectors from other perspectives. The process of using a cross-attention mechanism to combine the query vector Q from the current perspective with the M key vectors and M value vectors from other perspectives to obtain an enhanced multi-view fusion output is as follows:

[0111] First, calculate the correlation between the query vector Q and all region features from other perspectives. This can be achieved by calculating the dot product of Q with each key vector (or other similarity calculation methods), resulting in a correlation matrix A.

[0112] From the relevance matrix A, select the M key vectors and their corresponding M value vectors that are most relevant to the query vector Q. This can be achieved by selecting the M largest values ​​in matrix A and using their indexes within the matrix to determine the most relevant keys and values.

[0113] For the selected M key vectors, a cross-attention mechanism is used to compute the attention weights between them and the query vector Q. Further similarity calculations (such as dot products) can be performed between the query vector and each key vector, and then these similarity values ​​are normalized using a softmax function to obtain a set of weights.

[0114] Finally, the calculated attention weights are used to perform a weighted summation of the M value vectors, resulting in an enhanced multi-view fusion output. This output integrates information from different perspectives, providing a richer and more accurate feature representation for the current perspective.

[0115] It should be noted that, in order to more effectively fuse image features from different perspectives, this embodiment introduces a fusion method based on a cross-attention mechanism. Before fusion, the first region feature and M region features are first linearly mapped, converting them into query vectors, key vectors, and value vectors. These vectors will be used for subsequent cross-attention calculations. By calculating the correlation between the query vector and each key vector, the N region features most relevant to the first region feature are selected, and their corresponding key vectors and value vectors are determined. Finally, the cross-attention mechanism is used to integrate these vectors to obtain an enhanced first region feature, and its shape is adjusted to match the original first region feature.

[0116] To further illustrate, consider two images: a frontal view of a face (first view) and a side view of a face (second view). This embodiment aims to fuse features from these two images for more accurate face recognition. First, a first region feature related to the eyes is extracted from the first view image and converted into a query vector. Then, multiple region features from the second view image are converted into key and value vectors. Next, the correlation between the query vector and each key vector is calculated, and the N region features most relevant to the eye features (e.g., a side view of the nose) are selected. Finally, a cross-attention mechanism is used to integrate the value vectors of these selected region features with the query vector, resulting in a region feature that enhances the eye features.

[0117] The embodiments provided in this application introduce a cross-attention mechanism for feature fusion, which can more effectively integrate image features from different perspectives. It not only considers the global correlation between features, but also enhances key features through local, fine-grained correlation calculation.

[0118] As an optional approach, local feature extraction is performed on the first-view image and the second-view image to obtain multiple first image features corresponding to the first-view image and multiple second image features corresponding to the second-view image. This includes: inputting the first-view image and the second-view image into a trained first image processing model to obtain multiple first image features and multiple second image features output by the trained first image processing model, wherein the first image processing model is used to extract the original features of the image.

[0119] The anomaly identification results of the detected object are obtained through the first fusion feature and the second fusion feature, including:

[0120] S5-1, Input the first fusion feature and the second fusion feature into the second image processing model to obtain the first target feature and the second target feature output by the second image processing model, wherein the second image processing model is used to process abnormal image features into normal image features;

[0121] S5-2, acquire multiple first image features and compare them with the first target features to obtain a first comparison result, wherein the first comparison result is used to indicate the feature differences between the multiple first image features and the first target features;

[0122] S5-3, acquire multiple second image features and compare them with the second target features to obtain a second comparison result, wherein the second comparison result is used to indicate the feature differences between the multiple second image features and the second target features;

[0123] S5-4. Based on the first comparison result and the second comparison result, obtain the anomaly identification result.

[0124] Optionally, in this embodiment, the first image processing model can be a model for extracting raw features from an image. Raw features may include, but are not limited to, edges, corners, textures, color distribution, etc., which are inherent, unprocessed features in the image.

[0125] Optionally, in this embodiment, the second image processing model can be a model for processing image features, capable of converting abnormal image features into normal image features. Here, "abnormal" and "normal" are relative to a predefined benchmark or normal state.

[0126] It should be noted that this embodiment describes an image processing flow. First, features are extracted from images from two different viewpoints using a first image processing model. Then, these features are fused and processed using a second image processing model to convert potentially anomalous image features into normal image features. Next, the original image features are compared with the processed target features to identify differences and determine whether anomalies exist.

[0127] To further illustrate, consider two photographs of the same object taken from different angles, and the goal is to detect any anomalies. First, a first image processing model extracts features from both photographs. Then, these features are fused and processed by a second image processing model, which can transform anomalous features into normal features. Finally, the original features are compared with the processed features; if a significant difference is found, it may indicate an anomaly in the original image.

[0128] The embodiments provided in this application enable the extraction of more comprehensive feature information from images viewed from two different perspectives, and the accurate identification of anomalies in the images through feature comparison. This method not only improves the sensitivity of anomaly detection but also enhances its accuracy. By applying two image processing models consecutively, anomalies in images can be processed and identified more effectively, providing a more reliable anomaly detection solution for various application scenarios.

[0129] As an optional approach, before inputting the first-view image and the second-view image into the first image processing model to obtain multiple first image features and multiple second image features output by the first image processing model, the method further includes:

[0130] S6-1, Obtain the first image processing model after training, and use it as the teacher model;

[0131] S6-2, Obtain the second image processing model to be trained, as the student model;

[0132] S6-3 uses multiple normal images as training samples and trains the student model through the teacher model to obtain the trained second image processing model.

[0133] Optionally, in this embodiment, the teacher model can be a pre-trained model with high performance and accuracy, used to guide the learning process of the student model.

[0134] Optionally, in this embodiment, the student model can be a model to be trained, with the goal of improving its own performance by mimicking the output of the teacher model.

[0135] Optionally, in this embodiment, multiple normal images are used as training samples, and a student model is trained through a teacher model to obtain a trained second image processing model. This can be understood as a knowledge distillation training method, which involves constructing a lightweight small model (student model) and using the supervision information of a larger model (teacher model) with better performance to train this small model.

[0136] It should be noted that, considering the limited number of negative samples in anomaly detection scenarios, which often cannot meet the needs of model training, this embodiment uses a pre-trained teacher model (i.e., the first image processing model) to train a student model (i.e., the second image processing model). This process uses only a few normal images as training samples to ensure that the student model can learn the ability to extract features from normal images. Even if negative samples (anomaly images) are lacking during model training, a model (the second image processing model) that can be used to process image features can still be trained.

[0137] To further illustrate, consider a hypothetical scenario with a pre-trained image processing model A (the teacher model) that accurately extracts features from normal images. The goal is to train a new model B (the student model) that can perform a similar task. To achieve this, a set of normal images can be used as common input to both the teacher and student models. For each image, the teacher model first processes it and outputs its features. These features are then used as the learning objective for the student model, and an optimization process is employed to make the student model's output as close as possible to the teacher model's output.

[0138] Through the embodiments provided in this application, the student model can inherit some of the capabilities of the teacher model, particularly its feature extraction capabilities in processing normal images. This not only accelerates the training process of the student model but also improves its accuracy and robustness when processing unseen normal images. Furthermore, since the student model is typically simpler than the teacher model, it may have faster inference speeds and lower computational costs, which is crucial for real-time applications or resource-constrained environments.

[0139] As an optional approach, the anomaly identification result of the detected object is obtained through the first fusion feature and the second fusion feature, including:

[0140] S7-1, using the first fusion feature, obtain the first anomaly score of the detected object from the first viewpoint, wherein the first anomaly score is used to indicate the possibility or degree of anomaly of the detected object from the first viewpoint;

[0141] S7-2, through the second fusion feature, obtain the second anomaly score of the detected object in the second view, wherein the second anomaly score is used to indicate the possibility or degree of anomaly of the detected object in the second view;

[0142] S7-3, Obtain the anomaly identification result based on the first anomaly score and the second anomaly score.

[0143] It should be noted that this embodiment describes a process in which anomaly identification results of a detected object are obtained using a first fusion feature from a first perspective and a second fusion feature from a second perspective. The specific steps include: calculating an anomaly score for the detected object from the first perspective using the first fusion feature; this score is used to assess the probability or degree of anomaly in the detected object from the first perspective; similarly, obtaining an anomaly score for the detected object from the second perspective using the second fusion feature to assess the abnormal state of the detected object from the second perspective. Finally, combining the anomaly scores from both perspectives, a comprehensive judgment is made to obtain the final anomaly identification result.

[0144] To further illustrate, consider the optional assumption that a quality inspection is being performed on an industrial part. Images of the part are captured from two different perspectives (front view and side view), and their respective fused features are extracted. For the front view image, a first anomaly score of 0.8 is calculated, indicating a high probability of anomaly in the part from this perspective. For the side view image, a second anomaly score of 0.5 is obtained, indicating that the part also exhibits some degree of anomaly from the side view perspective. By combining these two scores, a comprehensive anomaly identification result can be obtained, for example, determining that the part is a defective product.

[0145] The embodiments provided in this application utilize fused features from multiple perspectives for comprehensive anomaly identification, which improves the accuracy and reliability of detection. Because information from different perspectives is considered, anomalies in the detected object can be captured more comprehensively. Furthermore, by quantifying anomaly scores, the degree of anomaly can be assessed in a finer-grained manner, providing richer information for subsequent decision-making or processing.

[0146] As an optional approach, the first anomaly score of the detected object from a first-view perspective is obtained through the first fusion feature, including:

[0147] S8-1, obtain the pixel-level anomaly score corresponding to each pixel position on the first view image through the first fusion feature, wherein the pixel-level anomaly score is used to indicate the possibility or degree of anomaly at the pixel position;

[0148] S8-2, the pixel-level anomaly score that indicates the highest probability or degree of an anomaly at the pixel location is determined as the first anomaly score.

[0149] It should be noted that this embodiment describes how to obtain the first anomaly score of the detected object from the first viewpoint using the first fusion feature. First, it uses the first fusion feature to calculate a pixel-level anomaly score for each pixel location. This score reflects the probability or degree of anomaly at that pixel location. Then, the largest pixel-level anomaly score is selected as the first anomaly score for the entire first viewpoint.

[0150] To further illustrate, consider an optional assumption: an image of an industrial part, and the first fusion feature of this image has already been obtained through some method. Next, a pixel-level anomaly score is calculated for each pixel in the image. For example, a relatively high pixel-level anomaly score at a specific pixel location might indicate the presence of a defect or anomaly at that location. Finally, the highest pixel-level anomaly score is selected as the first anomaly score for this image.

[0151] The embodiments provided in this application calculate pixel-level anomaly scores for each pixel and select the highest score as the first anomaly score, which can more accurately locate the locations where anomalies may exist in the image.

[0152] As an alternative approach, multiple normal images are used as training samples to train a student model using a teacher model, resulting in a trained second image processing model, including:

[0153] Perform the following steps until a trained second image processing model is obtained:

[0154] S9-1, determine the current normal image from multiple normal images, and use the current normal image as the current training sample;

[0155] S9-2, input the current training sample into the trained first image processing model, and the trained first image processing model performs multi-scale feature encoding on the current training sample to obtain the K first initial features output by the trained first image processing model at K scales;

[0156] S9-3, Local feature extraction is performed on the K initial features to obtain multi-view image features at each of the K scales;

[0157] S9-4, fuse the multi-view image features at each scale to obtain the multi-view fused features at each of the K scales;

[0158] S9-5 integrates the multi-view fusion features at each scale to obtain multi-scale integrated features;

[0159] S9-6, given the current second image processing model to be trained, the multi-scale integrated features are input into the current second image processing model, and the current second image processing model performs multi-scale feature decoding on the multi-scale integrated features to obtain the K second initial features output by the current second image processing model at K scales;

[0160] S9-7, obtain the mean square error among the initial features at the same scale among the K first initial features and K second initial features, and obtain K mean square errors;

[0161] S9-8, sum the K mean square errors to obtain the loss value;

[0162] S9-9: If the loss value does not meet the convergence condition, determine the next normal image from multiple normal images and use the next normal image as the current training sample.

[0163] S9-10, if the loss value meets the convergence condition, the current second image processing model is determined as the trained second image processing model.

[0164] Optionally, in this embodiment, multi-scale feature encoding / decoding can be understood as the process of extracting or generating image features at different scales (or resolutions), which helps to understand image content from both global and local levels. Here, K is only for illustrative purposes and can be understood as an integer greater than 1. Local feature extraction can be understood as extracting features of specific regions or details from an image, such as edges, corners, or textures.

[0165] Optionally, in this embodiment, after obtaining the multi-view fusion features at each scale, the next step is to further integrate these fusion features from different scales. Integration is different from fusion: fusion combines features from different perspectives within the same scale, while integration combines features that have already been fused across scales.

[0166] Furthermore, the purpose of integration is to aggregate information from different scales into a unified feature representation. Features at different scales contain a variety of information about the image, from local to global. By integrating these features, the model can simultaneously consider the details and overall structure of the image, thereby making more accurate judgments or predictions.

[0167] Specifically, integration methods can include feature concatenation, attention-based weighted summation, or the use of more complex deep learning structures (such as convolutional neural networks, recurrent neural networks, etc.) to automatically learn and combine features at different scales. The final result is a multi-scale integrated feature that fuses information from all scales, providing a rich feature base for subsequent image processing and analysis.

[0168] It should be noted that this embodiment first uses a trained first image processing model (possibly a teacher model) to encode multi-scale features of normal images. Then, these features are locally extracted, fused, and integrated to generate multi-scale integrated features. These integrated features are then input into a second image processing model (student model) to be trained for multi-scale feature decoding. The loss value is obtained by calculating the mean squared error between the initial features of the first and second models at the same scale. This loss value is used to evaluate the performance of the current second image processing model. If the convergence condition is not met, iterative optimization continues; if it is met, the current model is determined to be the trained model.

[0169] To further illustrate, consider 100 normal images as training samples for the teacher model. First, select one image and encode its multi-scale features using a first image processing model. These features are then processed and input into a second image processing model. The loss value is obtained by calculating the mean squared error between the features of the two models at the same scale. If this loss value exceeds a set threshold, select the next image and perform the same processing until the loss value meets the model's convergence condition.

[0170] The embodiments provided in this application provide a trained second image processing model that can effectively simulate the multi-scale feature encoding capabilities of the first image processing model (the teacher model). This means that for a new input image, the second image processing model will be able to generate multi-scale feature representations similar to those of the teacher model, thereby exhibiting good performance in various image processing tasks.

[0171] As an optional approach, for ease of understanding, the above-mentioned multi-view anomaly identification method is implemented through a multi-view defect detection (MVAD) framework, as follows: Figure 4 As shown, the system consists of a teacher (encoder) model with a reverse knowledge distillation structure, a student (decoder) model, and three intermediate multi-view fusion modules at different scales. The intermediate layers perform multi-view feature fusion on features extracted from the pre-trained teacher model at three different scales. Specifically, for each scale, there are several Multi-View Automatic Selection (MVAS) modules with the same number of channels. Each single-view feature, after being encoded with a position, enters the MVAS module (e.g., MVAS1, MVAS2, MVAS3) for feature fusion with other multi-view features. The multi-view fused features at each scale are then fused together through a feature pyramid-like structure and finally fed to the student model for decoding. Finally, the loss function of the feature maps at each scale of the teacher and student models is calculated and summed.

[0172] To further illustrate, alternatively, for example... Figure 4 As shown, the multi-view defect detection framework consists of a teacher-student model with a reverse knowledge distillation structure and three intermediate multi-view fusion modules at different scales. The intermediate layers perform multi-view feature fusion on features extracted from the pre-trained teacher model at the three different scales. Specifically, for each scale j, there exists a corresponding MVAS with the same number of channels. j Module. After adding the positional encoding P to each single-view feature, layer normalization LN is first performed. j Then enter the MVAS module, for example:

[0173]

[0174]

[0175] Features after multi-view fusion at each scale The data will be fused together using a feature pyramid-like structure and ultimately fed into the student model. Finally, the encoder... and decoder The mean squared error of the feature maps at each scale is calculated and summed. Specifically:

[0176]

[0177] Where J is the number of feature stages used in the experiment.

[0178] It should be noted that, compared to the traditional defect detection input of n∈N batches of images, the multi-view task input is measured in units of samples, with each input consisting of p∈N samples, where each sample contains images from v∈N viewpoints. Therefore, the actual input batch size is p×v∈N. During the training phase, for each viewpoint feature X... s ∈R p×c×h×w All of them must be combined with the features X from the other four perspectives. m ∈R p×(v-1)c×h×w Perform feature fusion to obtain enhanced features for the current viewpoint. During the testing phase, the defect map S was obtained. px ∈R pv×H×W The pixel-level anomaly score is used. The maximum value across the entire defect image is taken as the image-level anomaly score S. im ∈R pv Finally, the image-level anomaly scores S for all views in each sample are calculated. im Take the maximum value as the anomaly score S at the sample level. sa ∈R p For the true value G at the sample level sa ∈R p The rule is set so that if any viewpoint in the sample contains an anomaly, then the sample is considered an anomalous sample. Conversely, if no anomalous region is found in any viewpoint of the sample, then the sample is considered a normal sample.

[0179] Optionally, in this embodiment, MVAS is applied to multi-view fusion at the feature scale, and the specific steps are as follows:

[0180] S10-1, First, for input X i ∈R v×h×w×c The multi-view feature map is generated, where v represents the number of views. First, this feature map is further processed into a non-overlapping region of size a×a. Each region contains There are 10 feature vectors. For any one of these viewpoint feature maps... Perform a linear mapping as a query, and then consider the features from other perspectives. Perform a linear mapping as keys and values.

[0181] Q s =X s W q ,K m =X m W k V m =X m W v ,

[0182] Among them, W q Wk W v ∈R c×c These are the weights of the linear mapping.

[0183] S10-2, then it is necessary to find the most relevant region between the current viewpoint feature region and the multi-viewpoint feature regions to obtain the correlation matrix. Specifically, in step S9-1, we obtain... Based on this, further processed regional features are obtained. Similarly, it can be obtained from From Subsequently, the correlation matrix It can be calculated using the formula:

[0184] A c =A s (A m ) K

[0185] S10-3, the correlation matrix A obtained in step S10-2 c This study reveals the semantic correlation between single-view and multi-view region features. After obtaining the correlation matrix, the goal is to calculate the K regions with the most semantically relevant and closely related multi-view region features for each single-view region feature. The ultimate objective is to obtain the most relevant index matrix.

[0186]

[0187] S10-4, the most relevant index matrix obtained in step S10-3 Based on this, the semantic feature similarity of each region in the current viewpoint to all regions in other viewpoints is automatically calculated, and the K most relevant regions are selected as the objects for subsequent global attention calculation. To calculate the attention between the feature map of a single viewpoint and the feature maps of the K most relevant regions in other viewpoints, it is first necessary to combine the multi-view feature keys and values... Through the most relevant index matrix Obtain the most relevant features K of the top K regions m, V m Cross-attention is applied to both queries and key-value pairs to achieve enhanced multi-view fusion output.

[0188]

[0189] S10-5, the enhanced single-view features obtained in step S10-4 should be... Adjust the shape to match the original input features to obtain the output single-view features. Iterate v times and concatenate the results from each viewpoint to obtain the final output Y. o ∈R v×h×w×c .

[0190] Optionally, the computational complexity Ω of cross-attention in this embodiment is less than or equal to the computational complexity Ω of single-view features as queries Q∈R. (hw)×c Input and multi-view features are used as key values ​​K, V∈R (vhw)×c Input is:

[0191] Ω (cross-attention) = 2v(hw) 2 c+v(hw) 2 =(2c+1)v(hw) 2 ≈O((hw) 2 ),

[0192] Where v represents the number of multi-viewpoints, and c represents the dimension of the features. Cross-attention has a highest quadratic complexity of O(hw). 2 Furthermore, it consumes a significant amount of GPU memory. Therefore, MVAS, with its lower computational complexity, was proposed in section 3.2.2. Its computational complexity mainly comprises two parts: automatic multi-view window selection and proximity-related cross-attention. The total computational complexity is:

[0193]

[0194] Here, 'a' represents the size of the nearest neighbor window, and 'k' represents the number of the most relevant nearest neighbor windows. The inequality between the arithmetic mean and the geometric mean is applied here. The equality holds only in this context:

[0195]

[0196] Therefore, MVAS can achieve a minimum linear complexity of O((hw)) by adjusting the size of the nearest neighbor window and the number of the most relevant nearest neighbor windows.

[0197] The embodiments provided in this application address the challenging task of multi-view defect detection. For feature fusion and learning across multiple views, an MVAS algorithm is proposed. Specifically, the feature map is first processed into neighboring attention windows. Then, the semantic relevance matrix of each window in a single view and all multi-view windows is calculated, and the top K most relevant indices are determined. Finally, a cross-attention mechanism is applied to each window in a single view and the K windows in the multi-view windows. The computational complexity of MVAS is analyzed to be linear at the minimum if an appropriate window size is chosen. The entire MVAS framework uses an encoder-decoder architecture and employs MVAS blocks of varying numbers and dimensions at each feature scale, fused together through convolutional operations in a feature pyramid structure.

[0198] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0199] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0200] According to another aspect of the embodiments of this application, a multi-view anomaly recognition apparatus for implementing the above-described multi-view anomaly recognition method is also provided. For example... Figure 5 As shown, the device includes:

[0201] The acquisition unit 502 is used to acquire a first-view image of the detected object from a first-view perspective and a second-view image of the detected object from a second-view perspective.

[0202] The extraction unit 504 is used to extract local features from the first viewpoint image and the second viewpoint image to obtain multiple first image features corresponding to the first viewpoint image and multiple second image features corresponding to the second viewpoint image.

[0203] The first fusion unit 506 is used to fuse multiple first image features and first related features to obtain a first fused feature, wherein the first related feature is a related feature among multiple second image features whose correlation with the multiple first image features is greater than or equal to a first threshold.

[0204] The second fusion unit 508 is used to fuse multiple second image features and second related features to obtain a second fused feature, wherein the second related feature is a related feature among multiple first image features whose correlation with multiple second image features is greater than or equal to a second threshold.

[0205] The identification unit 510 is used to obtain the abnormal identification result of the detected object through the first fusion feature and the second fusion feature.

[0206] For specific implementation examples, please refer to the examples shown in the above-described multi-view anomaly recognition method; these examples will not be repeated here.

[0207] As an optional solution, the first fusion unit 506 includes:

[0208] The processing module is used to process multiple first image features into first region features and second region features, and to process multiple second image features into M region features, where M is an integer greater than 1;

[0209] The first fusion module is used to fuse the first relevant region feature and the first region feature to obtain a new first region feature, wherein the first relevant region feature is a region feature among M region features that has a correlation degree with the first region feature greater than or equal to the first threshold.

[0210] The second fusion module is used to fuse the second related region feature and the second region feature to obtain a new second region feature, wherein the second related region feature is a region feature among M region features whose correlation with the second region feature is greater than or equal to the first threshold.

[0211] The combination module is used to combine the new first region feature and the new second region feature to obtain the first fused feature.

[0212] For specific implementation examples, please refer to the examples shown in the above-described multi-view anomaly recognition method; these examples will not be repeated here.

[0213] As an optional solution, the device also includes:

[0214] The first related module is used to calculate the correlation matrix between the first regional feature and each of the M regional features before fusing the first related regional feature and the first regional feature to obtain a new first regional feature. The correlation matrix is ​​used to represent the semantic correlation between the first regional feature and each of the M regional features.

[0215] The second related module is used to determine the top N regional features in terms of semantic relevance from M regional features based on the relevance matrix before fusing the first related regional features and the first regional features to obtain a new first regional feature, where N is an integer greater than 0.

[0216] For specific implementation examples, please refer to the examples shown in the above-described multi-view anomaly recognition method; these examples will not be repeated here.

[0217] As an optional solution, the device also includes:

[0218] The first mapping module is used to perform a linear mapping on the first region features before fusing the first relevant region features and the first region features to obtain new first region features, and to convert the first region features into a query vector.

[0219] The second mapping module is used to perform linear mapping on the M regional features respectively before fusing the first relevant regional features and the first regional features to obtain new first regional features, so as to obtain M key vectors and M value vectors.

[0220] The device also includes:

[0221] The first vector module is used to determine the N key vectors corresponding to the N regional features from the M key vectors after determining the top N regional features in terms of semantic relevance from the M regional features based on the relevance matrix.

[0222] The second vector module is used to determine the N value vectors corresponding to the N regional features from the M value vectors after determining the top N regional features in terms of semantic relevance from the M regional features based on the relevance matrix.

[0223] The first fusion module includes:

[0224] The first fusion submodule is used to integrate the query vector, N key vectors, and N value vectors using a cross-attention mechanism to obtain enhanced first region features;

[0225] The second fusion submodule is used to adjust the enhanced first region feature to the original shape of the first region feature to obtain a new first region feature.

[0226] For specific implementation examples, please refer to the examples shown in the above-described multi-view anomaly recognition method; these examples will not be repeated here.

[0227] As an optional solution, the extraction unit 504 includes: a first model module, used to input a first viewpoint image and a second viewpoint image into a first image processing model to obtain multiple first image features and multiple second image features output by the first image processing model, wherein the first image processing model is used to extract the original features of the image;

[0228] The identification unit 510 includes:

[0229] The second model module is used to input the first fusion feature and the second fusion feature into the second image processing model to obtain the first target feature and the second target feature output by the second image processing model. The second image processing model is used to process abnormal image features into normal image features.

[0230] The first acquisition module is used to acquire multiple first image features and compare them with the first target features to obtain a first comparison result, wherein the first comparison result is used to indicate the feature differences between the multiple first image features and the first target features;

[0231] The second acquisition module is used to acquire multiple second image features and compare them with the second target features to obtain a second comparison result, wherein the second comparison result is used to indicate the feature differences between the multiple second image features and the second target features;

[0232] The third acquisition module is used to acquire the anomaly identification result based on the first comparison result and the second comparison result.

[0233] For specific implementation examples, please refer to the examples shown in the above-described multi-view anomaly recognition method; these examples will not be repeated here.

[0234] As an optional solution, the device also includes:

[0235] The fourth acquisition module is used to acquire the trained first image processing model as a teacher model before inputting the first view image and the second view image into the first image processing model to obtain multiple first image features and multiple second image features output by the first image processing model.

[0236] The fifth acquisition module is used to acquire the second image processing model to be trained as a student model before inputting the first view image and the second view image into the first image processing model to obtain multiple first image features and multiple second image features output by the first image processing model.

[0237] The training module is used to train the student model using multiple normal images as training samples before inputting the first-view image and the second-view image into the first image processing model to obtain multiple first image features and multiple second image features output by the first image processing model, so as to obtain the trained second image processing model.

[0238] For specific implementation examples, please refer to the examples shown in the above-described multi-view anomaly recognition method; these examples will not be repeated here.

[0239] As an optional approach, the training module includes: an execution submodule, used to perform the following steps until a trained second image processing model is obtained: determining the current normal image from multiple normal images and using the current normal image as the current training sample; inputting the current training sample into the trained first image processing model, and having the trained first image processing model perform multi-scale feature encoding on the current training sample to obtain K first initial features output by the trained first image processing model at K scales; performing local feature extraction on the K first initial features respectively to obtain multi-view image features at each of the K scales; fusing the multi-view image features at each scale to obtain multi-view fused features at each of the K scales; and integrating the multi-view fused features at each scale to obtain multi-scale integrated features. The process involves several steps: First, given a current second image processing model to be trained, the multi-scale integrated features are input into the current second image processing model. The current second image processing model then decodes the multi-scale integrated features at multiple scales, obtaining K second initial features output by the current second image processing model at K scales. Next, the mean squared errors between the initial features at the same scale among the K first initial features and K second initial features are obtained, resulting in K mean squared errors. These K mean squared errors are then summed to obtain the loss value. If the loss value does not meet the convergence condition, the next normal image is determined from multiple normal images and used as the current training sample. If the loss value meets the convergence condition, the current second image processing model is determined as the trained second image processing model.

[0240] For specific implementation examples, please refer to the example shown in the multi-view anomaly recognition method, which will not be repeated here.

[0241] As an optional solution, the identification unit 510 includes:

[0242] The sixth acquisition module is used to acquire the first anomaly score of the detected object from the first viewpoint through the first fusion feature, wherein the first anomaly score is used to indicate the possibility or degree of anomaly of the detected object from the first viewpoint.

[0243] The seventh acquisition module is used to acquire the second anomaly score of the detected object from the second perspective through the second fusion feature, wherein the second anomaly score is used to indicate the probability or degree of anomaly of the detected object from the second perspective;

[0244] The ninth acquisition module is used to obtain the anomaly identification result based on the first anomaly score and the second anomaly score.

[0245] For specific implementation examples, please refer to the examples shown in the above-described multi-view anomaly recognition method; these examples will not be repeated here.

[0246] As an optional solution, the sixth acquisition module includes:

[0247] The first acquisition submodule is used to acquire pixel-level anomaly scores corresponding to each pixel position on the first viewpoint image through the first fusion feature, wherein the pixel-level anomaly score is used to indicate the possibility or degree of anomaly at the pixel position;

[0248] The second acquisition submodule is used to determine the pixel-level anomaly score with the highest probability or degree of anomaly at the indicated pixel location as the first anomaly score.

[0249] For specific implementation examples, please refer to the examples shown in the above-described multi-view anomaly recognition method; these examples will not be repeated here.

[0250] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described multi-view-based anomaly recognition method is also provided. This electronic device may, but is not limited to, [providing a multi-view-based anomaly recognition method]. Figure 1 The user equipment 102 or server 112 shown in the figure, in this embodiment, is taken as an example of an electronic device, namely user equipment 102. Further, as shown in the figure... Figure 6 As shown, the electronic device includes a memory 602 and a processor 604. The memory 602 stores a computer program, and the processor 604 is configured to execute the steps in any of the above method embodiments via the computer program.

[0251] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0252] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0253] S1, acquire the first-view image of the detected object from the first perspective and the second-view image of the detected object from the second perspective;

[0254] S2, perform local feature extraction on the first viewpoint image and the second viewpoint image to obtain multiple first image features corresponding to the first viewpoint image and multiple second image features corresponding to the second viewpoint image;

[0255] S3, fuse multiple first image features and first related features to obtain a first fused feature, wherein the first related feature is a related feature among multiple second image features whose correlation with multiple first image features is greater than or equal to a first threshold;

[0256] S4, fuse multiple second image features and second related features to obtain a second fused feature, wherein the second related feature is a related feature among multiple first image features whose correlation with multiple second image features is greater than or equal to a second threshold;

[0257] S5. Obtain the anomaly identification result of the detected object through the first fusion feature and the second fusion feature.

[0258] Alternatively, as those skilled in the art will understand, Figure 6 The structure shown is for illustrative purposes only. Figure 6 This does not limit the structure of the aforementioned electronic devices. For example, the electronic device may also include components that are more... Figure 6 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 6 The different configurations shown.

[0259] The memory 602 can be used to store software programs and modules, such as the program instructions / modules corresponding to the multi-view anomaly recognition method and device in this embodiment. The processor 604 executes various functional applications and data processing by running the software programs and modules stored in the memory 602, thereby realizing the aforementioned multi-view anomaly recognition method. The memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 602 may further include memory remotely located relative to the processor 604, and these remote memories can be connected to electronic devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 602 may be used, but is not limited to, to store information such as first-view images, second-view images, and anomaly recognition results. As an example, such as... Figure 6 As shown, the memory 602 may include, but is not limited to, the acquisition unit 502, extraction unit 504, first fusion unit 506, second fusion unit 508, and identification unit 510 in the multi-view anomaly recognition device. Furthermore, it may include, but is not limited to, other module units in the multi-view anomaly recognition device, which will not be elaborated upon in this example.

[0260] Optionally, the transmission device 606 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 606 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 606 is a radio frequency (RF) module, used for wireless communication with the Internet.

[0261] In addition, the above-mentioned electronic device also includes: a display 608 for displaying the above-mentioned first-view image, second-view image and anomaly recognition results and other information; and a connection bus 610 for connecting the various module components in the above-mentioned electronic device.

[0262] In other embodiments, the aforementioned user equipment or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer network, and any form of computing device, such as a server, user equipment, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0263] According to one aspect of this application, a computer program product is provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions provided in embodiments of this application.

[0264] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0265] It should be noted that the computer system of the electronic device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0266] A computer system includes a Central Processing Unit (CPU), which performs various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) or loaded from RAM. ROM also stores various programs and data required for system operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output interfaces (I / O interfaces) are also connected to the bus.

[0267] The following components are connected to the input / output interface: input sections including keyboards, mice, etc.; output sections including cathode ray tubes (CRTs), liquid crystal displays (LCDs), and speakers; storage sections including hard drives; and communication sections including network interface cards such as LAN cards and modems. The communication section performs communication processing via a network such as the Internet. Drives are also connected to the input / output interface as needed. Removable media, such as disks, optical discs, magneto-optical discs, semiconductor memories, etc., are installed on the drive as needed so that computer programs read from them can be installed into the storage section as required.

[0268] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions defined in the system of this application.

[0269] According to one aspect of this application, a computer-readable storage medium is provided, wherein a processor of a computer device reads computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0270] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0271] S1, acquire the first-view image of the detected object from the first perspective and the second-view image of the detected object from the second perspective;

[0272] S2, perform local feature extraction on the first viewpoint image and the second viewpoint image to obtain multiple first image features corresponding to the first viewpoint image and multiple second image features corresponding to the second viewpoint image;

[0273] S3, fuse multiple first image features and first related features to obtain a first fused feature, wherein the first related feature is a related feature among multiple second image features whose correlation with multiple first image features is greater than or equal to a first threshold;

[0274] S4, fuse multiple second image features and second related features to obtain a second fused feature, wherein the second related feature is a related feature among multiple first image features whose correlation with multiple second image features is greater than or equal to a second threshold;

[0275] S5. Obtain the anomaly identification result of the detected object through the first fusion feature and the second fusion feature.

[0276] Optionally, in embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0277] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware of an electronic device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0278] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0279] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.

[0280] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0281] In the several embodiments provided in this application, it should be understood that the disclosed user equipment can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the processing of a unit is only one logical function. In actual implementation, there may be other processing methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0282] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0283] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0284] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A multi-view anomaly identification method, characterized in that, include: Acquire a first-view image of the detected object from a first-view perspective, and a second-view image of the detected object from a second-view perspective; Local feature extraction is performed on the first viewpoint image and the second viewpoint image to obtain multiple first image features corresponding to the first viewpoint image and multiple second image features corresponding to the second viewpoint image; The plurality of first image features and first related features are fused to obtain a first fused feature, wherein the first related feature is a related feature among the plurality of second image features whose correlation with the plurality of first image features is greater than or equal to a first threshold; The plurality of second image features and the second related features are fused to obtain a second fused feature, wherein the second related feature is a related feature among the plurality of first image features whose correlation with the plurality of second image features is greater than or equal to a second threshold; The anomaly identification result of the detected object is obtained by using the first fusion feature and the second fusion feature.

2. The method according to claim 1, characterized in that, The plurality of first image features include first region features and second region features, and the plurality of second image feature processing includes M region features, where M is an integer greater than 1; The step of fusing the plurality of first image features and first related features to obtain a first fused feature includes: The first relevant region feature and the first region feature are fused to obtain a new first region feature, wherein the first relevant region feature is a region feature among the M region features whose correlation with the first region feature is greater than or equal to the first threshold. The second relevant region feature and the second region feature are fused to obtain a new second region feature, wherein the second relevant region feature is a region feature among the M region features whose correlation with the second region feature is greater than or equal to the first threshold. The new first region feature and the new second region feature are combined to obtain the first fused feature.

3. The method according to claim 2, characterized in that, Before fusing the first relevant region feature and the first region feature to obtain a new first region feature, the method further includes: Calculate the correlation matrix between the first regional feature and each of the M regional features, wherein the correlation matrix is ​​used to represent the semantic correlation between the first regional feature and each of the M regional features; Based on the correlation matrix, the top N region features with the highest semantic relevance are determined from the M region features, where N is an integer greater than 0.

4. The method according to claim 3, characterized in that, Before fusing the first relevant region feature and the first region feature to obtain a new first region feature, the method further includes: Perform a linear mapping on the features of the first region to convert the features of the first region into a query vector; Linear mapping is performed on the M region features respectively to obtain M key vectors and M value vectors; After determining the top N region features in terms of semantic relevance from the M region features based on the relevance matrix, the method further includes: From the M key vectors, determine the N key vectors corresponding to the N region features; From the M value vectors, determine the N value vectors corresponding to the N regional features; The step of fusing the first relevant region features and the first region features to obtain a new first region feature includes: The query vector, N key vectors, and N value vectors are integrated using a cross-attention mechanism to obtain the new first region feature.

5. The method according to claim 1, characterized in that, The step of extracting local features from the first viewpoint image and the second viewpoint image to obtain multiple first image features corresponding to the first viewpoint image and multiple second image features corresponding to the second viewpoint image includes: inputting the first viewpoint image and the second viewpoint image into a first image processing model to obtain the multiple first image features and the multiple second image features output by the first image processing model; The step of obtaining the anomaly identification result of the detected object through the first fusion feature and the second fusion feature includes: The first fusion feature and the second fusion feature are input into the second image processing model to obtain the first target feature and the second target feature output by the second image processing model, wherein the second image processing model is used to process abnormal image features into normal image features; The plurality of first image features are obtained and compared with the first target feature to obtain a first comparison result, wherein the first comparison result is used to indicate the feature differences between the plurality of first image features and the first target feature; The plurality of second image features are obtained and compared with the second target features to obtain a second comparison result, wherein the second comparison result is used to indicate the feature differences between the plurality of second image features and the second target features; The anomaly identification result is obtained based on the first comparison result and the second comparison result.

6. The method according to claim 5, characterized in that, Before inputting the first viewpoint image and the second viewpoint image into the first image processing model to obtain the plurality of first image features and the plurality of second image features output by the first image processing model, the method further includes: Obtain the first image processing model after training, and use it as the teacher model; Obtain the second image processing model to be trained, and use it as the student model; Using multiple normal images as training samples, the student model is trained through the teacher model to obtain the trained second image processing model.

7. The method according to claim 6, characterized in that, The process of using multiple normal images as training samples to train the student model through the teacher model to obtain the trained second image processing model includes: Perform the following steps until the trained second image processing model is obtained: The current normal image is determined from the plurality of normal images, and the current normal image is used as the current training sample; The current training sample is input into the trained first image processing model, and the trained first image processing model performs multi-scale feature encoding on the current training sample to obtain K first initial features output by the trained first image processing model at K scales; Local feature extraction is performed on the K initial features to obtain multi-view image features at each of the K scales; The multi-view image features at each scale are fused to obtain the multi-view fused features at each of the K scales; The multi-view fusion features at each scale are integrated to obtain multi-scale integrated features; Given a current second image processing model to be trained, the multi-scale integrated features are input into the current second image processing model, and the current second image processing model performs multi-scale feature decoding on the multi-scale integrated features to obtain K second initial features output by the current second image processing model at the K scales. The mean square error between the initial features at the same scale among the K first initial features and the K second initial features is obtained to obtain K mean square errors; The loss value is obtained by summing the K mean square errors. If the loss value does not meet the convergence condition, the next normal image is determined from the plurality of normal images, and the next normal image is used as the current training sample; If the loss value satisfies the convergence condition, the current second image processing model is determined as the trained second image processing model.

8. The method according to claim 1, characterized in that, The step of obtaining the anomaly identification result of the detected object through the first fusion feature and the second fusion feature includes: The first fusion feature is used to obtain the first anomaly score of the detected object from the first perspective, wherein the first anomaly score is used to indicate the possibility or degree of anomaly of the detected object from the first perspective. The second fusion feature is used to obtain the second anomaly score of the detected object from the second perspective, wherein the second anomaly score is used to indicate the probability or degree of anomaly of the detected object from the second perspective. The anomaly identification result is obtained based on the first anomaly score and the second anomaly score.

9. The method according to claim 8, characterized in that, The step of obtaining the first anomaly score of the detected object from a first perspective through the first fusion feature includes: The pixel-level anomaly score is obtained for each pixel position on the first viewpoint image through the first fusion feature, wherein the pixel-level anomaly score is used to indicate the possibility or degree of anomaly at the pixel position; The pixel-level anomaly score that indicates the highest probability or degree of an anomaly at the pixel location is determined as the first anomaly score.

10. An anomaly detection device based on multiple perspectives, characterized in that, include: The acquisition unit is used to acquire a first-view image of the detected object from a first-view perspective and a second-view image of the detected object from a second-view perspective. The extraction unit is used to extract local features from the first viewpoint image and the second viewpoint image to obtain multiple first image features corresponding to the first viewpoint image and multiple second image features corresponding to the second viewpoint image. The first fusion unit is used to fuse the plurality of first image features and the first related features to obtain a first fused feature, wherein the first related feature is a related feature among the plurality of second image features whose correlation with the plurality of first image features is greater than or equal to a first threshold; The second fusion unit is used to fuse the plurality of second image features and the second related features to obtain a second fused feature, wherein the second related feature is a related feature among the plurality of first image features whose correlation with the plurality of second image features is greater than or equal to a second threshold; The identification unit is used to obtain the anomaly identification result of the detected object through the first fusion feature and the second fusion feature.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program is executed by an electronic device to perform the method according to any one of claims 1 to 9.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 9.

13. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 9 through the computer program.

Images