Home appliance component material detection method and device based on multi-modal image fusion and medium

Abstract

The application discloses a household appliance component material quality detection method and device based on multi-modal image fusion and a medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a visible light image, a dual-energy X-ray image and an actual measured weight of a same household appliance object, and performing registration on the visible light image and the dual-energy X-ray image; extracting surface features of the registered visible light image, and identifying an existing state, a specification and a projection density distribution of an internal core component by using the registered dual-energy X-ray image; establishing a whole machine weight constraint inversion mechanism, calculating initial material properties and an initial weight of the internal core component by using the projection density distribution, and correcting the initial material properties and the initial weight by using the actual measured weight to obtain a target material proportion and a target weight proportion of each internal core component relative to the actual measured weight of the whole machine; and comprehensively generating an evaluation report corresponding to the household appliance object by using the surface features, the existing state, the specification, the target material proportion and the target weight proportion of the internal core component.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus and medium for detecting the material of home appliance components based on multimodal image fusion. Background Technology

[0002] The recycling of waste electronic and electrical equipment has become an important part of the circular economy. Waste household appliances (such as refrigerators and washing machines) contain core components like motors and compressors, which have high recycling value. However, the complex internal structure of these appliances and the presence of overlapping components make accurate material analysis and value assessment challenging.

[0003] Currently, the inspection of discarded household appliances mainly relies on manual experience or single-modal inspection techniques. Traditional visual inspection can only acquire the external condition of the appliance and cannot see the true condition of the internal core components; for example, it is common to find compressors that are intact on the outside but have had high-value internal components removed. To overcome visual blind spots, some companies have introduced X-ray transmission (XRT) technology for internal imaging. However, when dealing with heavy objects like household appliances with densely packed components, the superposition of different materials can produce strong grayscale confusion, making it easy to misjudge and inaccurately estimate weight when relying solely on image density to infer materials. In addition, existing recycling lines usually treat visual inspection, X-ray scanning, and whole-machine weighing as independent workstations. There is a lack of spatiotemporal correlation between multi-source data, and the whole-machine weighing data is only used for simple warehousing records and is not involved in the calculation process for internal material analysis.

[0004] In summary, existing detection methods, due to isolated multi-source data and a lack of global constraints in computation, cannot accurately assess internal material properties and weight proportions in complex occlusion environments. This not only increases the trial-and-error costs of blind disassembly but also easily leads to distorted recycling value predictions, affecting the formulation of subsequent sorting strategies. Therefore, there is an urgent need for a detection solution that can deeply integrate multi-dimensional information and accurately identify component materials. Summary of the Invention

[0005] This invention provides a method, device, and medium for detecting the material of home appliance components based on multimodal image fusion, which solves the problem of material misjudgment and estimation distortion caused by isolated multi-source data in the prior art under complex occlusion.

[0006] In a first aspect, embodiments of the present invention provide a method for detecting the material of household appliance components based on multimodal image fusion, comprising: Acquire visible light images, dual-energy X-ray images, and the actual weight of the whole appliance for the same appliance, and register the visible light images and dual-energy X-ray images; The surface features of the registered visible light image are extracted, and the presence, specifications and projection density distribution of the internal core components are identified using the registered dual-energy X-ray image. A whole machine weight constraint inversion mechanism is established. The initial material properties and initial weight of the internal core components are calculated using the projection density distribution. The initial material properties and initial weight are corrected by the actual weight of the whole machine to obtain the target material ratio and target weight ratio of each internal core component relative to the actual weight of the whole machine. An evaluation report is generated based on the comprehensive surface characteristics, state of existence, specifications, proportion of target material, and proportion of target weight of the appliance.

[0007] In one alternative implementation, the presence, size, and projection density distribution of internal core components are identified using registered dual-energy X-ray images, including: Logarithmic subtraction is performed on the high-energy and low-energy images contained in the registered dual-energy X-ray images to obtain the material attenuation feature map. A preset target detection model was used to extract features from the registered dual-energy X-ray image and material attenuation feature map, and the internal structure with plastic coating and metal parts were segmented as candidate components. Based on the segmentation results, candidate components are identified as internal core components and their bounding box positions are obtained. The existence status is determined based on the bounding box positions, and the physical size of the bounding box is calculated by combining the preset benchmark magnification to determine the specifications. The projection density distribution is calculated based on the pixel grayscale values ​​within the bounding box positions.

[0008] In one optional implementation, a whole-machine weight constraint inversion mechanism is established. The initial material properties and initial weight of the internal core components are calculated using the projected density distribution. These initial material properties and initial weights are then corrected using the actual measured weight of the whole machine to obtain the target material proportion and target weight proportion of each internal core component relative to the actual measured weight of the whole machine, including: Based on the projected density distribution and the preset material attenuation coefficient table, the initial volume of each internal core component is calculated by introducing the preset density compensation coefficient, and the initial material properties are determined by combining the preset material identification confidence level. Based on the initial volume and initial material properties, the initial weight of each internal core component is calculated, and the estimated weight of the first whole machine is obtained by summarizing them. Calculate the weight difference between the estimated weight of the first whole machine and the actual measured weight of the whole machine. When the weight difference exceeds the preset error threshold, trigger the weight constraint inversion iteration. In the weight constraint inversion iteration, the actual weight of the whole machine is used as the optimization target. The confidence level of material identification and the density compensation coefficient are adjusted. Based on the adjusted confidence level of material identification and the adjusted density compensation coefficient, the updated estimated weight of the whole machine is calculated until the error between the updated estimated weight of the whole machine and the actual weight of the whole machine is not greater than the preset error threshold. The final material properties and final weight of each internal core component relative to the actual weight of the whole machine at the time of iteration convergence are used as the target material ratio and target weight ratio, respectively.

[0009] In one alternative implementation, the weight constraint inversion iteration also includes: When the actual weight of the whole machine is lower than the estimated weight of the first whole machine, and the difference between the two exceeds the preset abnormal judgment threshold, the preset standard specification database is queried in combination with the existence status and specifications to obtain the standard internal weight corresponding to each internal core component. The standard internal weight is compared with the corresponding initial weight. If the difference exceeds the preset tolerance range, it is determined that there is a missing critical component in the internal core component. Based on the missing status of key components, the corresponding initial material properties and initial weight are reduced or reset to zero and updated. The weight constraint inversion iteration is then performed based on the updated initial material properties and initial weight, and the missing status of key components is output as the basis for generating subsequent evaluation reports.

[0010] In one optional implementation, surface features include information on surface soiling status, and the assessment report includes missing item warnings, component ratios, recycling levels, and valuation ranges. By comprehensively considering surface features, state of existence, specifications, target material percentage, and target weight percentage, an assessment report corresponding to the appliance is generated, including: Internal features are extracted from the registered dual-energy X-ray images. The internal features include at least the state of existence and the size. The surface features and internal features are then input into a pre-defined multimodal fusion network. Based on the surface dirt status information, the fusion weights of surface features and internal features are dynamically calculated, and the surface features and internal features are fused based on the fusion weights to obtain fused features; wherein, when the surface dirt status information indicates that the degree of surface damage exceeds the preset damage threshold or there is dirt coverage, the fusion weight of surface features is reduced and the fusion weight of internal features is increased. Based on fusion characteristics, target material proportion, and target weight proportion, the potential content of valuable materials is predicted; Based on the potential content and state of existence of valuable materials, a deficiency warning, component ratio, recovery level, and valuation range are determined to generate an assessment report.

[0011] In one optional implementation, after generating the evaluation report corresponding to the home appliance object, the following is also included: Based on the recycling level and valuation range in the assessment report, the target diversion channel and corresponding sorting trajectory parameters corresponding to the home appliance are matched from the preset warehouse diversion strategy library. Generate tag information containing the target diversion channel and valuation range, and control the labeling device connected to the communication connection to attach the tag information to the home appliance object; The sorting trajectory parameters are sent to the actuator connected to the communication link to drive the actuator to move the tagged home appliance objects to the corresponding classification storage area.

[0012] In one alternative implementation, after the actuator moves the tagged home appliance objects to the corresponding category storage area, it also includes: Receive the actual material and weight data of the corresponding home appliance object from the downstream physical verification node, and convert them into the actual material percentage and actual weight percentage. Compare the actual material percentage and actual weight percentage with the corresponding target material percentage and target weight percentage for consistency. Using the error feedback data generated by the comparison, the basic parameters inside the whole machine weight constraint inversion mechanism are adaptively fine-tuned and updated. The basic parameters include the material attenuation coefficient table or density compensation coefficient.

[0013] In one alternative implementation, the internal core components include at least one of a motor, a compressor, and a printed circuit board within the appliance object; Extracting surface features from the registered visible light image includes: using a pre-trained convolutional neural network to perform feature mapping on the registered visible light image, and obtaining feature vectors representing the brand parameters and physical integrity of the home appliance object as surface features.

[0014] Secondly, embodiments of the present invention provide a material detection device for home appliance components based on multimodal image fusion, comprising: The image acquisition and registration module is used to acquire visible light images, dual-energy X-ray images, and the actual measured weight of the whole appliance for the same appliance, and to register the visible light images and dual-energy X-ray images. The feature extraction and penetration recognition module is used to extract the surface features of the registered visible light image and to use the registered dual-energy X-ray image to identify the existence, specifications and projection density distribution of the internal core components. The weight constraint inversion correction module is used to establish a whole machine weight constraint inversion mechanism. It calculates the initial material properties and initial weight of the internal core components using the projection density distribution, and corrects the initial material properties and initial weight using the actual weight of the whole machine to obtain the target material ratio and target weight ratio of each internal core component relative to the actual weight of the whole machine. The evaluation generation module is used to generate an evaluation report for the home appliance object by comprehensively considering surface characteristics, state of existence, specifications, target material ratio, and target weight ratio.

[0015] A third aspect of this invention provides an electronic device, which includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention.

[0017] Beneficial Effects: This invention provides a method, apparatus, device, and medium for detecting the material properties of home appliance components based on multimodal image fusion. By registering visible light images with dual-energy X-ray transmission images, it achieves integrated and collaborative identification of the internal and external states of home appliances. It overcomes the limitations of isolated multi-source data in traditional methods by constructing a whole-machine weight constraint inversion mechanism. The measured weight of the entire machine is used as the core technical constraint to invert and correct the initial material properties and weight calculated from image density. This mechanism effectively overcomes the defects of grayscale confusion and material misjudgment caused by overlapping and occlusion of components in existing technologies, improving the calculation accuracy of the material and weight ratio of each internal core component. The evaluation report generated based on multi-dimensional features and inversion results can objectively reflect the true composition and value of the home appliance. It can output evaluation result data for backend sorting and dismantling decisions, which helps reduce the cost of blind dismantling. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the steps of a method for detecting the material of home appliance components based on multimodal image fusion, as provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the functional modules of a home appliance component material detection device based on multimodal image fusion provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] The present invention will be further described below with reference to the accompanying drawings.

[0021] Reference Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0022] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (Wi-Fi). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0023] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0024] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and computer programs.

[0025] exist Figure 1In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the home appliance component material detection device based on multimodal image fusion stored in the memory 1005 through the processor 1001, and executes the home appliance component material detection method based on multimodal image fusion provided in the embodiment of the present invention.

[0026] In the recycling and harmless treatment process of Waste Electrical and Electronic Equipment (WEEE), discarded household appliances (such as refrigerators, washing machines, and air conditioners) face challenges due to their complex internal structures, highly heterogeneous materials, and the tendency for non-standard replacement or unauthorized disassembly of parts during long-term use or disposal. Methods relying solely on surface visual features cannot penetrate the appliance's casing to obtain the true condition of its internal core components. Furthermore, using single-energy XRT imaging technology, especially with thick, multi-layered objects like appliances, results in grayscale confusion due to the overlapping materials, leading to material misjudgment and distorted weight estimation. In addition, existing weighing data collection in production lines often serves only as the final data recording node, resulting in isolated data silos between the collected physical weight and the front-end image analysis data, failing to achieve effective technological synergy.

[0027] To overcome the aforementioned technical deficiencies, Embodiment 1 of this application provides a method for detecting the material of household appliance components based on multimodal image fusion. For example... Figure 2 As shown, it includes the following steps: S101: Acquire visible light image, dual-energy X-ray image and actual weight of the same appliance, and register the visible light image and dual-energy X-ray image.

[0028] Specifically, step S101 above includes the following sub-steps: S1011, Acquire Visible Light Images. Responding to the arrival signal triggered by the photoelectric switch, the execution unit simultaneously acquires high-resolution visible light images of the appliance under uniform, ring-shaped shadowless light illumination using multiple industrial-grade red-green-blue (RGB) color array cameras deployed above and to the side of the detection platform. To eliminate image color differences and vignetting caused by lighting fluctuations in the production environment, the execution unit performs white balance correction, adaptive histogram equalization, and Gaussian filtering-based denoising preprocessing on the acquired visible light images. Furthermore, the execution unit uses an image stitching algorithm to fuse the multi-view visible light images into a panoramic view, obtaining visible light images that comprehensively characterize the appliance's physical outline, color space distribution, surface nameplate markings, and surface physical damage and contamination.

[0029] S1012, Acquiring Dual-Energy X-ray Images. The execution unit acquires dual-energy X-ray images of household appliances using a dual-energy X-ray emission source deployed on one side of the detection platform and a dual-layer linear array detector array on the opposite side. Dual-energy X-ray imaging technology simultaneously emits and receives two different energy spectra of X-ray photons: high energy (e.g., tube voltage set to 140kV) and low energy (e.g., tube voltage set to 80kV). These photons penetrate the metal or engineering plastic casing of the household appliance to obtain a perspective projection of its internal structure. The front detector in the dual-layer array (typically using a scintillator material sensitive to low-energy photons) primarily absorbs low-energy X-rays, while the rear detector, after physical filtering with copper or aluminum sheets (used to filter and absorb low-energy photons), primarily absorbs high-energy X-rays. The execution unit performs dark-field correction and flat-field correction on the raw signals acquired by the detectors to eliminate inherent dead pixels and gain unevenness noise from the sensors, thereby simultaneously outputting high-quality dual-energy X-ray image data containing both high-energy and low-energy images in a single scan.

[0030] S1013, Acquire the actual weight of the entire appliance. During the stable transport of the appliance through the inspection platform via the conveyor belt, the actuator uses a high-precision dynamic weighing sensor array (such as a piezoelectric or strain gauge weighing module with a high-frequency sampling rate) mounted at its bottom to acquire the actual weight of the appliance in real time. To overcome mechanical vibration interference in the industrial environment, the actuator performs low-pass filtering on the weight signal at the front-end hardware level and employs a Kalman filter algorithm at the software calculation level to perform state estimation and smoothing / denoising of the continuous weight signal sequence. Simultaneously, the actuator combines zero-point automatic tracking technology to remove the tare weight of the conveyor belt, recording the stabilized absolute net weight value and binding it to a unique identification code (ID) generated for the current appliance.

[0031] S1014, Spatiotemporal registration is performed on the visible light image and the dual-energy X-ray image. The execution unit uses an edge detection operator to extract the physical edge contour of the appliance in the visible light image and the outermost contour bounding rectangle in the dual-energy X-ray image, respectively. The execution unit extracts feature points in the images and uses a registration algorithm based on a combination of perspective transformation and affine transformation calculated in advance using a physical calibration board (such as a checkerboard with lead markings) to scale, translate, and correct distortion in the visible light image, so that the outer contour of the appliance body in the two images coincides in the same pixel coordinate system. At the same time, the execution unit uses a high-precision hardware timestamp synchronization mechanism generated by a programmable logic controller (PLC) to establish a data buffer queue, achieving strict synchronization between the registered multimodal image and the weighing data on the time slice, ensuring that the subsequently extracted fusion features all originate from the same physical entity.

[0032] S102, extract the surface features of the registered visible light image, and use the registered dual-energy X-ray image to identify the presence, specifications and projection density distribution of the internal core components.

[0033] In one alternative implementation, the internal core components include at least one of a motor, a compressor, and a printed circuit board (PCB) within the appliance object; extracting surface features from the registered visible light image includes: performing feature mapping on the registered visible light image using a pre-trained convolutional neural network to obtain feature vectors characterizing the brand parameters and physical integrity of the appliance object as surface features.

[0034] Specifically, the above implementation method includes the following sub-steps: S1021, the execution entity normalizes the pre-processed and registered visible light image and inputs it into a pre-trained deep convolutional neural network (CNN) architecture. In the offline stage, the CNN uses a training dataset containing a large number of images of the appearance of discarded household appliances and their corresponding brands, dirt and integrity labels. It performs supervised learning training by minimizing the cross-entropy loss function through the backpropagation algorithm until the network weights converge to a steady state.

[0035] S1022, the execution entity extracts local texture, geometric edges and color gradient features of the image layer by layer through convolutional kernels, batch normalization layers and nonlinear activation functions in the network.

[0036] S1023, the executing entity extracts a one-dimensional feature vector in the fully connected layer or global average pooling layer of the network. This feature vector is used to characterize the brand parameters of the home appliance (such as brand logo features, model nameplate style) and the external physical integrity (such as shell dents, holes, rust or covering conditions) as surface features.

[0037] In one alternative implementation, the presence, size, and projection density distribution of internal core components are identified using registered dual-energy X-ray images, including: Logarithmic subtraction is performed on the high-energy and low-energy images contained in the registered dual-energy X-ray images to obtain the material attenuation feature map. A preset target detection model was used to extract features from the registered dual-energy X-ray image and material attenuation feature map, and the internal structure with plastic coating and metal parts were segmented as candidate components. Based on the segmentation results, candidate components are identified as internal core components and their bounding box positions are obtained. The existence status is determined based on the bounding box positions, and the physical size of the bounding box is calculated by combining the preset benchmark magnification to determine the specifications. The projection density distribution is calculated based on the pixel grayscale values ​​within the bounding box positions.

[0038] Specifically, the above implementation method includes the following sub-steps: S1024, the executing entity acquires the high-energy transmission grayscale value and low-energy transmission grayscale value corresponding to each pixel in the dual-energy X-ray image, and calculates their natural logarithmic values ​​respectively. The executing entity subtracts or calculates the ratio of the low-energy logarithmic value and the high-energy logarithmic value according to a polynomial model to generate a two-dimensional material attenuation feature map.

[0039] In step S1025, the execution entity concatenates the original dual-energy X-ray image with the material attenuation feature map along the channel dimension to form a multi-channel input tensor, which is then fed into the pre-set target detection model. The target detection model is trained by optimizing parameters by minimizing the bounding box regression loss and the category prediction loss using a sample dataset containing X-ray images of various types of discarded household appliances, internal component category labels, and corresponding real bounding box locations. The execution entity extracts multi-scale features through a Feature Pyramid Network (FPN), segments out the enclosed internal components and directly exposed metal parts, and uniformly extracts and labels them as candidate parts.

[0040] S1026, the executing entity uses a model classifier to perform category regression on candidate components, identifies the internal core component, and outputs the coordinates of the two-dimensional bounding box of the component in the image pixel coordinate system, confirming the existence of the core component. The executing entity combines the reference magnification matrix of the X-ray imaging equipment and the pixel size of the bounding box to calculate the actual physical geometric dimensions of the component as its specifications through geometric mapping. The executing entity extracts all pixel-level grayscale values ​​within the two-dimensional bounding box region in the material attenuation feature map, generates a two-dimensional density matrix, and forms a projected density distribution.

[0041] S103, establish a whole machine weight constraint inversion mechanism, calculate the initial material properties and initial weight of the internal core components using the projection density distribution, and correct the initial material properties and initial weight using the actual weight of the whole machine to obtain the target material ratio and target weight ratio of each internal core component relative to the actual weight of the whole machine.

[0042] In one optional implementation, a whole-machine weight constraint inversion mechanism is established. The initial material properties and initial weight of the internal core components are calculated using the projected density distribution. These initial material properties and initial weights are then corrected using the actual measured weight of the whole machine to obtain the target material proportion and target weight proportion of each internal core component relative to the actual measured weight of the whole machine, including: Based on the projected density distribution and the preset material attenuation coefficient table, the initial volume of each internal core component is calculated by introducing the preset density compensation coefficient, and the initial material properties are determined by combining the preset material identification confidence level. Based on the initial volume and initial material properties, the initial weight of each internal core component is calculated, and the estimated weight of the first whole machine is obtained by summarizing them. Calculate the weight difference between the estimated weight of the first whole machine and the actual measured weight of the whole machine. When the weight difference exceeds the preset error threshold, trigger the weight constraint inversion iteration. In the weight constraint inversion iteration, the actual weight of the whole machine is used as the optimization target. The confidence level of material identification and the density compensation coefficient are adjusted. Based on the adjusted confidence level of material identification and the adjusted density compensation coefficient, the updated estimated weight of the whole machine is calculated until the error between the updated estimated weight of the whole machine and the actual weight of the whole machine is not greater than the preset error threshold. The final material properties and final weight of each internal core component relative to the actual weight of the whole machine at the time of iteration convergence are used as the target material ratio and target weight ratio, respectively.

[0043] Specifically, the above implementation method includes the following sub-steps: S1031, the executing entity maps the acquired projected density distribution of each component to a preset material attenuation coefficient table to estimate the equivalent thickness distribution matrix. The executing entity introduces a preset density compensation coefficient to spatially correct the equivalent thickness matrix, integrating and summing over the two-dimensional projected area corresponding to the bounding box to obtain the initial volume. The executing entity combines the probability distribution vectors of various materials output by the target detection model, assigns a preset material recognition confidence level, and determines the initial material properties.

[0044] S1032, the executing entity extracts the normal standard physical density corresponding to the initial material properties, calculates the weighted mixed density using the material identification confidence level as a weight, and then multiplies it with the initial volume to obtain the initial weight of each internal core component. The executing entity sums up the initial weights of all core components detected this time, adds the estimated reference weight of the shell and auxiliary materials, and summarizes them to obtain the first estimated weight of the whole machine.

[0045] S1033, the execution entity calculates the weight difference between the estimated weight of the first complete machine and the actual measured weight of the complete machine. If the weight difference exceeds the preset error threshold, the execution entity triggers the weight constraint inversion iteration.

[0046] In S1034, during the weight constraint inversion iteration, the executing agent constructs an optimization objective function model. The error loss objective function based on the L1 norm is formulated as follows:

[0047] In the formula, This represents the objective function value for minimizing the overall weight estimation error. This indicates the actual measured weight of the entire machine; This indicates the total number of internal core components; Indicates the first The basic initial volume of each internal core component; Indicates the first Density compensation coefficient for each component; Indicates the total number of preset basic material types; Indicates the first The component is composed of the first The confidence level of material identification for each material, while satisfying the constraints. ; Indicates the first The standard physical density constant of a material; This represents the reference weight of the outer shell and non-core auxiliary materials, estimated based on the outer contour area and preset shell thickness of the home appliance. The material identification confidence level is... The range of values ​​is limited to Its value is determined based on the normalized probability distribution characteristics of the Softmax classification layer output of the front-end object detection model; the density compensation coefficient The range of values ​​is limited to The value is based on the statistical distribution of a large number of waste household appliance samples: due to the irregular shape of the internal core components and the presence of front and rear obstruction, the equivalent thickness will produce a fluctuation error of about 30% in the upper and lower averages when integrating in two dimensions. Therefore, limiting it within this range can prevent divergence and volume distortion that violates physical common sense during inversion iteration.

[0048] S1035, The execution entity minimizes the objective function value. To optimize the objective, Particle Swarm Optimization (PSO) or gradient descent is employed, within the aforementioned parameter value range. and Adjusting density compensation coefficient within physical boundary constraints Material recognition confidence The value of is substituted into the model to calculate the updated estimated weight of the entire machine. Until the error... If the error does not exceed a preset error threshold, the iteration converges. The executing entity extracts the final material properties and final weight of each component at this point, divides them by the normalized weight of the entire machine, and outputs the target material percentage and target weight percentage.

[0049] In one alternative implementation, the weight constraint inversion iteration also includes: When the actual weight of the whole machine is lower than the estimated weight of the first whole machine, and the difference between the two exceeds the preset abnormal judgment threshold, the preset standard specification database is queried in combination with the existence status and specifications to obtain the standard internal weight corresponding to each internal core component. The standard internal weight is compared with the corresponding initial weight. If the difference exceeds the preset tolerance range, it is determined that there is a missing critical component in the internal core component. Based on the missing status of key components, the corresponding initial material properties and initial weight are reduced or reset to zero and updated. The weight constraint inversion iteration is then performed based on the updated initial material properties and initial weight, and the missing status of key components is output as the basis for generating subsequent evaluation reports.

[0050] Specifically, the above implementation method includes the following sub-steps: S1036 When the actual weight of the whole machine is lower than the estimated weight of the first whole machine, and the negative difference ratio exceeds the preset abnormal judgment threshold, the execution subject pauses the regular iteration, extracts the specification information of the core components, and retrieves the standard internal weight of the model in the standard specification database.

[0051] S1037, the executing entity compares the standard internal weight with the initial weight. If the difference exceeds the preset tolerance range, it is determined that the core of the internal core component has a physical defect and is marked as being in a critical component missing state.

[0052] S1038, for the marked missing components, the executing entity reduces or resets their initial material properties and corresponding initial weight parameters. The executing entity re-executes the weight constraint inversion iteration with the updated parameters and stores the missing status of key components in local storage or a cloud database as input for subsequent evaluation report generation.

[0053] S104, taking into account surface characteristics, state of existence, specifications, target material ratio, and target weight ratio, generates an evaluation report for the corresponding home appliance object.

[0054] In one optional implementation, surface features include information on surface soiling status, and the assessment report includes missing item warnings, component ratios, recycling levels, and valuation ranges. By comprehensively considering surface features, state of existence, specifications, target material percentage, and target weight percentage, an assessment report corresponding to the appliance is generated, including: Internal features are extracted from the registered dual-energy X-ray images. The internal features include at least the state of existence and the size. The surface features and internal features are then input into a pre-defined multimodal fusion network. Based on the surface dirt status information, the fusion weights of surface features and internal features are dynamically calculated, and the surface features and internal features are fused based on the fusion weights to obtain fused features; wherein, when the surface dirt status information indicates that the degree of surface damage exceeds the preset damage threshold or there is dirt coverage, the fusion weight of surface features is reduced and the fusion weight of internal features is increased. Based on fusion characteristics, target material proportion, and target weight proportion, the potential content of valuable materials is predicted; Based on the potential content and state of existence of valuable materials, a deficiency warning, component ratio, recovery level, and valuation range are determined to generate an assessment report.

[0055] Specifically, the above implementation method includes the following sub-steps: In step S1041, the executing agent flattens and reduces the dimensions of the identified existence states, specifications, and density distributions into internal feature tensors of a uniform scale. The executing agent then feeds the extracted surface feature tensors and these internal feature tensors synchronously into a multimodal fusion network based on a cross-attention mechanism. This multimodal fusion network utilizes a sample dataset containing historical appliance appearance surface features, internal structural features, and corresponding effective fusion weights. It learns cross-modal feature mapping relationships through offline training by minimizing the cross-modal feature alignment loss through a joint optimization of mean squared error loss and cross-modal contrastive loss.

[0056] In step S1042, the executing entity extracts surface dirt state information from the surface features and inputs it into a pre-set dirt assessment classifier for scoring. When the damage level exceeds a preset damage threshold or dirt coverage exists, the executing entity dynamically reduces the fusion weights assigned to the surface features while increasing the fusion weights assigned to the internal features. The executing entity uses a multi-layer perceptron (MLP) to concatenate and map the dynamically weighted features, outputting fused features.

[0057] S1043, the executing entity inputs the fusion features, target material proportion, and target weight proportion into the lightweight regression network; the lightweight regression network is pre-trained using historical home appliance fusion feature data and labels of the actual content proportion of valuable materials obtained from downstream real physical dismantling, and the mean square error or mean absolute error is used as the loss function for parameter optimization during training. The executing entity accurately predicts the actual obtainable potential content proportion of valuable materials through this lightweight regression network.

[0058] S1044: The executing entity determines whether there is an abnormal missing core component based on the input status; if so, it activates a missing component warning. The executing entity converts the potential content of valuable materials into component proportions; it classifies the recycling level according to a hierarchical value classification tree algorithm; the executing entity obtains bulk metal spot market price data published by an external data server through a network communication interface and calculates the valuation range. The executing entity packages all data according to a preset format specification and generates an assessment report.

[0059] In one optional implementation, after generating the evaluation report corresponding to the home appliance object, the following is also included: Based on the recycling level and valuation range in the assessment report, the target diversion channel and corresponding sorting trajectory parameters corresponding to the home appliance are matched from the preset warehouse diversion strategy library. Generate tag information containing the target diversion channel and valuation range, and control the labeling device connected to the communication connection to attach the tag information to the home appliance object; The sorting trajectory parameters are sent to the actuator connected to the communication link to drive the actuator to move the tagged home appliance objects to the corresponding classification storage area.

[0060] Specifically, the above implementation method includes the following sub-steps: S1051, the executing entity matches the target sorting channel in the pre-set warehouse sorting strategy relational database based on the recovery level and valuation range. The strategy database synchronously returns the sorting trajectory parameters bound to that channel, which are pre-calculated by the inverse kinematics equations.

[0061] S1052, the executing entity generates a QR code or Radio Frequency Identification (RFID) electronic tag data instruction based on the target diversion channel number and valuation range. The executing entity then controls an automated labeling machine or laser marking machine connected to the communication link to attach the tag information to the outer surface of the appliance.

[0062] S1053, the executing entity sends the sorting trajectory parameters to the external executing mechanism such as the high-speed parallel sorting robot or articulated robotic arm connected to the communication via the industrial real-time Ethernet bus protocol, driving it to move the home appliances to the classified storage area.

[0063] In one alternative implementation, after the actuator moves the tagged home appliance objects to the corresponding category storage area, it also includes: Receive the actual material and weight data of the corresponding home appliance object from the downstream physical verification node, and convert them into the actual material percentage and actual weight percentage. Compare the actual material percentage and actual weight percentage with the corresponding target material percentage and target weight percentage for consistency. Using the error feedback data generated by the comparison, the basic parameters inside the whole machine weight constraint inversion mechanism are adaptively fine-tuned and updated. The basic parameters include the material attenuation coefficient table or density compensation coefficient.

[0064] Specifically, the above implementation method includes the following sub-steps: S1061, the executing entity receives the actual material composition and actual weight data transmitted back from the downstream physical verification node via the Industrial Internet of Things (IIoT). The executing entity extracts the measured total weight, divides the absolute weight of each material by the total weight, and converts it into the actual material proportion and actual weight proportion in a dimensionless manner.

[0065] S1062, the executing entity aligns and compares the converted actual proportion with the predicted target proportion, calculates and extracts the variance or residual value, and forms error feedback data.

[0066] In S1063, the executing entity uses the error feedback data as the gradient error term in the deep learning backpropagation process. Using this error gradient, the executing entity smoothly and adaptively fine-tunes the underlying parameters (such as the underlying mapping curve of the density compensation coefficient) within the overall weight constraint inversion mechanism through an adaptive learning rate optimization algorithm (such as the Adam optimizer).

[0067] This invention also provides a device for detecting the material of home appliance components based on multimodal image fusion, referring to... Figure 3 The diagram shows a functional block diagram of a home appliance component material detection device 300 based on multimodal image fusion according to the present invention. The device may include the following modules: The image acquisition and registration module 301 is used to acquire visible light images, dual-energy X-ray images and the actual weight of the whole appliance for the same appliance, and to register the visible light images and dual-energy X-ray images. The feature extraction and penetration recognition module 302 is used to extract the surface features of the registered visible light image and use the registered dual-energy X-ray image to identify the existence, specifications and projection density distribution of the internal core components. The weight constraint inversion correction module 303 is used to establish a whole machine weight constraint inversion mechanism. It calculates the initial material properties and initial weight of the internal core components using the projection density distribution, and corrects the initial material properties and initial weight using the actual weight of the whole machine to obtain the target material ratio and target weight ratio of each internal core component relative to the actual weight of the whole machine. The evaluation generation module 304 is used to generate an evaluation report for the home appliance object by comprehensively considering surface characteristics, state of existence, specifications, target material ratio, and target weight ratio.

[0068] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; The processor, when executing the program stored in the memory, implements the method for detecting the material of home appliance components based on multimodal image fusion of the present invention.

[0069] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM), or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0070] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0071] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for detecting the material of home appliance components based on multimodal image fusion, as described in the embodiments of the present invention.

[0072] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for material detection of household appliance components based on multimodal image fusion, characterized in that, include: Acquire visible light images, dual-energy X-ray images, and the actual measured weight of the same household appliance, and register the visible light images and the dual-energy X-ray images; The surface features of the registered visible light image are extracted, and the presence, specifications and projection density distribution of the internal core components are identified using the registered dual-energy X-ray image. A whole machine weight constraint inversion mechanism is established. The initial material properties and initial weight of the internal core components are calculated using the projection density distribution. The initial material properties and initial weight are corrected by the actual weight of the whole machine to obtain the target material ratio and target weight ratio of each internal core component relative to the actual weight of the whole machine. An evaluation report is generated based on the surface features, the state of existence, the specifications, the proportion of the target material, and the proportion of the target weight.

2. The method for detecting the material of household appliance components based on multimodal image fusion according to claim 1, characterized in that, The method of identifying the presence, specifications, and projection density distribution of internal core components using the registered dual-energy X-ray image includes: Logarithmic subtraction is performed on the high-energy and low-energy images contained in the registered dual-energy X-ray image to obtain a material attenuation feature map. A preset target detection model is used to extract features from the registered dual-energy X-ray image and the material attenuation feature map, and to segment the internal structure with plastic coating and the metal parts as candidate components. Based on the segmentation results, the candidate component is identified as the internal core component and its bounding box position is obtained. The existence state is determined based on the bounding box position. The physical size of the bounding box is calculated by combining it with a preset reference magnification factor to determine the specification. The projection density distribution is calculated based on the pixel grayscale values ​​within the bounding box position.

3. The method for detecting the material of household appliance components based on multimodal image fusion according to claim 1, characterized in that, The establishment of a whole-machine weight constraint inversion mechanism involves calculating the initial material properties and initial weight of the internal core components using the projected density distribution, and correcting the initial material properties and initial weight using the actual weight of the whole machine to obtain the target material ratio and target weight ratio of each internal core component relative to the actual weight of the whole machine, including: Based on the projected density distribution and the preset material attenuation coefficient table, a preset density compensation coefficient is introduced to calculate the initial volume of each of the internal core components, and the initial material properties are determined by combining the preset material identification confidence level. Based on the initial volume and the initial material properties, the initial weight of each of the internal core components is calculated, and the weight of the first complete machine is obtained by summing them up. Calculate the weight difference between the estimated weight of the first complete machine and the measured weight of the complete machine. When the weight difference exceeds a preset error threshold, trigger the weight constraint inversion iteration. In the weight constraint inversion iteration, the actual weight of the whole machine is used as the optimization target. The material identification confidence and the density compensation coefficient are adjusted, and the updated estimated weight of the whole machine is calculated based on the adjusted material identification confidence and the adjusted density compensation coefficient until the error between the updated estimated weight of the whole machine and the actual weight of the whole machine is not greater than the preset error threshold. The final material properties and final weight of each internal core component relative to the actual weight of the whole machine at the time of iteration convergence are respectively used as the target material ratio and the target weight ratio.

4. The method for detecting the material of household appliance components based on multimodal image fusion according to claim 3, characterized in that, The weight constraint inversion iteration also includes: When the actual weight of the whole machine is lower than the estimated weight of the first whole machine, and the difference between the two exceeds the preset abnormal judgment threshold, the preset standard specification database is queried in combination with the existence status and the specifications to obtain the standard internal weight corresponding to each of the internal core components. The standard internal weight is compared with the corresponding initial weight. If the difference exceeds the preset tolerance range, it is determined that the internal core component is missing a key component. Based on the missing status of the key component, the corresponding initial material properties and initial weight are reduced or reset to zero and updated, so as to continue to perform the weight constraint inversion iteration based on the updated initial material properties and initial weight, and the missing status of the key component is output as the basis for generating the evaluation report.

5. The method for detecting the material of household appliance components based on multimodal image fusion according to claim 1, characterized in that, The surface features include information on surface soiling status; the assessment report includes missing item warnings, component ratios, recycling levels, and valuation ranges; the assessment report for the appliance is generated by comprehensively considering the surface features, the state of existence, the specifications, the proportion of the target material, and the proportion of the target weight, including: Internal features are extracted based on the registered dual-energy X-ray image. The internal features include at least the existence state and the specifications. The surface features and the internal features are then input into a preset multimodal fusion network. Based on the surface dirt state information, the fusion weight of the surface features and the internal features is dynamically calculated, and the surface features and the internal features are fused based on the fusion weight to obtain a fused feature; wherein, when the surface dirt state information indicates that the degree of surface damage exceeds a preset damage threshold or there is dirt coverage, the fusion weight of the surface features is reduced and the fusion weight of the internal features is increased. Based on the fusion features, the target material proportion, and the target weight proportion, the potential content of valuable materials is predicted; Based on the potential content and state of existence of the valuable materials, the missing warning, the component ratio, the recovery level, and the valuation range are determined to generate the assessment report.

6. The method for material detection of household appliance components based on multimodal image fusion according to claim 5, characterized in that, After generating the evaluation report corresponding to the home appliance object, the process also includes: Based on the recycling level and valuation range in the assessment report, the target diversion channel and corresponding sorting trajectory parameters corresponding to the home appliance are matched from the preset warehouse diversion strategy library; Generate tag information containing the target diversion channel and the valuation range, and control the marking device connected to the communication connection to attach the tag information to the home appliance object; The sorting trajectory parameters are sent to the execution mechanism connected to the communication link to drive the execution mechanism to move the tagged home appliance object to the corresponding classification storage area.

7. The method for material detection of household appliance components based on multimodal image fusion according to claim 6, characterized in that, After the actuator moves the tagged home appliance to the corresponding category storage area, the method further includes: Receive the actual material and weight data of the corresponding home appliance object from the downstream physical verification node, and convert them into the actual material percentage and actual weight percentage. The actual material percentage and the actual weight percentage are compared with the corresponding target material percentage and the target weight percentage for consistency. Using the error feedback data generated by the comparison, the basic parameters inside the whole machine weight constraint inversion mechanism are adaptively fine-tuned and updated. The basic parameters include a material attenuation coefficient table or a density compensation coefficient.

8. The method for detecting the material of household appliance components based on multimodal image fusion according to claim 1, characterized in that, The internal core components include at least one of the following: a motor, a compressor, and a printed circuit board within the home appliance. The step of extracting the surface features of the registered visible light image includes: using a pre-trained convolutional neural network to perform feature mapping on the registered visible light image, and obtaining feature vectors representing the brand parameters and physical integrity of the home appliance object as the surface features.

9. A material detection device for household appliance components based on multimodal image fusion, characterized in that, include: The image acquisition and registration module is used to acquire visible light images, dual-energy X-ray images, and the actual measured weight of the whole appliance object, and to register the visible light images and the dual-energy X-ray images. The feature extraction and penetration recognition module is used to extract the surface features of the registered visible light image and to use the registered dual-energy X-ray image to identify the presence, size and projection density distribution of the internal core components. The weight constraint inversion correction module is used to establish a whole machine weight constraint inversion mechanism. It uses the projection density distribution to calculate the initial material properties and initial weight of the internal core components, and corrects the initial material properties and initial weight by the actual weight of the whole machine to obtain the target material ratio and target weight ratio of each internal core component relative to the actual weight of the whole machine. The evaluation generation module is used to generate an evaluation report for the home appliance object by comprehensively considering the surface features, the state of existence, the specifications, the proportion of the target material, and the proportion of the target weight.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for detecting the material of home appliance components based on multimodal image fusion as described in any one of claims 1 to 8.

Images