Material identification method and electronic equipment

Abstract

The invention relates to the technical field of intelligent equipment, and relates to an object surface material identification method, which is applied to electronic equipment and comprises the following steps: acquiring first image data of a to-be-identified object; dividing a plurality of material areas included in the to-be-identified object based on the first image data; determining millimeter wave data of each material area from the millimeter wave data of the object to be identified; and obtaining a first wave material type of each material region based on the millimeter wave data of each material region. And material recognition is performed on the to-be-recognized object through various recognition devices, so that the accuracy of material recognition is further improved. According to the technical scheme, the identification equipment is arranged on the mobile electronic equipment, and the mobile electronic equipment is controlled to move in the environment, so that material identification on a plurality of to-be-identified objects in the environment or larger to-be-identified surfaces of the to-be-identified objects is realized, manual placement of the identification equipment in the environment is avoided, and the detection cost is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of smart devices, and in particular to a material identification method and an electronic device. Background Art

[0002] With the development of wireless networks, more and more wireless signal transmission devices (such as wireless routers, etc.) will be deployed in business (To Business, ToB) scenarios, such as offices, workshops, restaurants, etc. For wireless signal transmission devices, it is crucial to provide good communication capabilities, that is, it is necessary to ensure that the wireless signal transmission devices have good transmission delay and large bandwidth during transmission.

[0003] The area corresponding to the ToB scenario is generally more complex. For example, the ToB scenario may include office furniture, production equipment, building partitions and materials, etc. Among them, the layout of various objects, such as the layout of partitions in the office, and the material of the objects (such as metal, concrete), etc., will affect the communication capability of the wireless signal transmission equipment. Therefore, network operators need to plan the location of wireless signal transmission equipment according to the actual conditions of various environments.

[0004] Figure 1 A schematic diagram of determining the location of wireless signal transmission equipment based on the layout of objects in the environment is shown. It can be seen that the network operator has established a wireless signal propagation model (i.e., a wireless signal propagation line schematic diagram) based on the layout of objects in the environment, and then determined the location of the wireless signal transmission equipment, thus realizing wireless network planning. However, Figure 1 In the above, the wireless signal transmission device is configured at position 100 based on the layout of various objects in the environment. It can be understood that the reflectivity, refractive index and penetration rate of different materials are different. Therefore, the material information of the object will also affect the transmission of the wireless signal. Therefore, the location of the wireless signal transmission device can be determined more accurately based on the layout of the object and the material information of the object. However, the existing technology lacks a solution that can accurately identify the material of an object. Summary of the invention

[0005] The present application provides a material identification method and medium for a detection device.

[0006] A first aspect provides a method for identifying the material of an object surface, which is applied to an electronic device, comprising:

[0007] Acquire first image data of the object to be identified; divide the object to be identified into a plurality of material regions based on the first image data; and determine the millimeter wave data of each material region from the millimeter wave data of the object to be identified;

[0008] Based on the millimeter wave data of each material area, the first wave material type of each material area is obtained.

[0009] In the present application, the electronic device here can be a control terminal, that is, a computer device that sends control instructions. Usually, the host computer can provide a user operation interactive interface and display feedback results to the user. The first image data here can be an electronic device controlling an image sensor, such as a multi-spectral camera, to capture an optical image of the surface to be identified of the object to be identified, and the optical image includes multi-channel spectral information. The material area here can be the material clustering result of the surface to be identified of the object to be identified, that is, the distribution area of ​​different types of materials forming the surface to be identified. The millimeter wave data here can be the echo data collected by the electronic device controlling the millimeter wave sensor to transmit millimeter waves to the object to be identified. The millimeter wave data of each material area can be determined by imaging the echo data to obtain the imaging result, and the imaging result can represent the echo data corresponding to the entire object to be identified or the surface to be identified of the object to be identified. Based on the distribution area of ​​different types of materials, the material type of each distribution area can be obtained by performing material recognition on the echo data corresponding to different distribution areas. The first wave material type here can be the material type determined based on the echo data of each distribution area.

[0010] It can be seen that by combining the multi-channel spectral information of the multi-spectral camera, multiple material regions on the object to be identified including multiple materials, that is, the distribution areas of multiple materials, can be determined, and then according to the millimeter wave data corresponding to each material region, the material type corresponding to each material region based on the millimeter wave data can be identified, further improving the accuracy of material identification.

[0011] In a possible implementation of the first aspect, the multiple material regions include a first material region and a second material region, wherein the first material region and the second material region have different colors or textures.

[0012] In the present application, the first image data, that is, the multi-channel spectral information, can be used to distinguish the distribution areas of different types of materials included in the object to be identified based on color or texture.

[0013] In a possible implementation of the first aspect, determining millimeter wave data of each material area from millimeter wave data of the object to be identified includes:

[0014] Based on the millimeter wave imaging algorithm, obtaining at least one of time information, distance information and angle information included in the millimeter wave data of the object to be identified;

[0015] Based on at least one of the time information, the distance information, and the angle information, confirming the detection position on the detected object to be identified corresponding to each millimeter wave data;

[0016] Based on the detection position of each millimeter wave data, the millimeter wave data belonging to each material area is divided.

[0017] In the present application, the millimeter wave imaging algorithm here can be to transform the millimeter wave data of the object to be identified, that is, the echo data, through Fourier transform frequency domain and time domain, filter the echo data, project the echo data to each pixel point or pixel area corresponding to the object to be identified, that is, the detection position on the object to be identified, and then delay compensate and coherently superimpose the echo data at each pixel point or pixel area to obtain the imaging result of the object to be identified based on the echo data.

[0018] In a possible implementation of the first aspect, obtaining the first wave material type of each material region based on the millimeter wave data of each material region includes:

[0019] The millimeter wave data of the first material region in each material region is matched with the millimeter wave data of the first material sample, and correspondingly, when the matching result satisfies the matching condition, it is determined that the first material region has the first wave material type of the first material sample.

[0020] In the present application, the matching here can be based on the electromagnetic scattering coefficient sequence corresponding to the millimeter wave data of the first material area. By matching with the electromagnetic scattering coefficient sequence of the known material, the material recognition result based on the millimeter wave data of the first material area can be obtained, that is, the first wave material type. In some embodiments, the matching method can include point matching of the sequence, or curve matching. Taking the point matching method as an example, the electromagnetic scattering coefficient sequence obtained by actual measurement is used to calculate the Euclidean distance between points with the electromagnetic scattering coefficient sequence of the corresponding angles of multiple materials in the sample library. The material corresponding to the electromagnetic scattering coefficient sequence with the smallest Euclidean distance in the sample library can be determined as the first wave material type.

[0021] In a possible implementation of the first aspect, obtaining the first wave material type of each material region based on the millimeter wave data of each material region further includes:

[0022] Determining that the first material region has a first image material type based on first image sub-data belonging to the first material region in the first image data;

[0023] A material type of a first material region is determined based on the first wave material type and the first image material type.

[0024] In the present application, the first image sub-data here can be a part of the image data in the first image data corresponding to the first material area, that is, a part of the multi-channel spectral information. The first image sub-data is judged by the spectral material decision model, and the first image material type based on the multi-channel spectral information is obtained according to the amplitude information of the multi-channel spectral information under different bands of light and the trend information of the connecting lines of points in different bands. The material type of the first material area obtained based on the first image sub-data of the first material area and the millimeter wave data of the first material area is the material recognition result.

[0025] In a possible implementation of the first aspect, determining the material type of the first material area based on the first wave material type and the first image material type further includes:

[0026] If the first wave material type probability corresponding to the first wave material type is lower than the first image material type probability corresponding to the first image material type, and the difference between the first wave material type probability and the first image material type probability exceeds a preset probability threshold, the first image material type probability is used to calibrate the first wave material type probability, and the material type of the first material area is determined according to the calibrated first wave material type probability.

[0027] In a possible implementation of the first aspect, calibrating the first wave of material type probabilities using the first image material type probability includes:

[0028] The first image material type probability and the first wave material type probability are averaged, the variance value or the mean square error value is calculated to obtain a calibrated first wave material type probability.

[0029] In the present application, the probability of the first wave material type corresponding to the first wave material type may be 0.51, and 0.51 indicates that 51% is metal, that is, the material recognition result based on millimeter wave data may be 51% metal. Relatively speaking, the probability of the first image material type corresponding to the first image material type may be 0.70, and 0.70 indicates that 70% is metal, that is, the material recognition result based on multi-channel spectral information may be 70% metal. If the preset probability threshold is 0.1, and the difference between 0.51 and 0.70 exceeds 0.1, 0.70 is used to calibrate 0.51. For example, the mean of 0.51 and 0.70 may be 0.60. It can be seen that by adjusting the material recognition result based on millimeter wave by the material recognition result based on multi-channel spectral information, the fused material recognition result of the object to be identified is more accurate.

[0030] In a possible implementation of the first aspect, determining the material type of the first material area based on the first wave material type and the first image material type further includes:

[0031] If the first wave material type probability corresponding to the first wave material type is higher than the first image material type probability corresponding to the first image material type, the material type of the first material area is determined according to the first wave material type probability.

[0032] In the present application, if the first wave material type probability corresponding to the first wave material type may be 0.70, relatively speaking, the first image material type probability corresponding to the first image material type may be 0.51, and the material type of the first material area may be 0.70.

[0033] In a possible implementation of the first aspect, obtaining first image data of the object to be identified includes:

[0034] Acquire multiple image data of the object to be identified taken from different angles;

[0035] Selecting first image data corresponding to a target angle from a plurality of image data;

[0036] Corresponding to the overexposure phenomenon of the first image data, the overexposure compensation is performed on the first image data based on the second image data among the multiple image data.

[0037] In the present application, the multiple image data here may be multi-channel spectral information of the object to be identified obtained at multiple different angles. The target angle here may be an angle selected from multiple different angles, for example, the target angle may be a normal view angle. Based on the angle difference between the normal view angle and each angle, the multi-channel spectral information of each angle outside the normal view angle may be rotated or translated, so that the multi-channel spectral information at different angles is unified to the normal view angle, the image data corresponding to the normal view angle may be the first image data, and the image data corresponding to multiple different angles outside the normal view angle may be the second image data. The overexposure phenomenon here may be an overexposed area in the first image data, that is, an overexposed highlight area in the first image data. From the second image data corresponding to multiple different angles outside the normal view angle, a local area corresponding to the overexposed area of ​​the first image data of the normal view angle is determined, that is, a correction area. The pixel values ​​of the pixels in the correction area are used to cover the overexposed area of ​​the first image data of the normal view angle to obtain the corrected first image data.

[0038] In a second aspect, an embodiment of the present application provides an electronic device, including:

[0039] a memory for storing instructions to be executed by one or more processors of the electronic device, and

[0040] The processor is one of the processors of the electronic device and is used to execute the material recognition method of the first aspect.

[0041] In a third aspect, an embodiment of the present application provides a computer program product, including: a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium containing a computer program code for executing the material identification method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of determining the position of a wireless signal transmission device according to the layout of objects in an environment provided by an embodiment of the present application;

[0043] Figure 2 A schematic diagram of an application scenario of material identification by a material identification system provided in an embodiment of the present application;

[0044] FIG. 3( a ) and FIG. 3( b ) are system architecture diagrams for implementing a material identification method provided in an embodiment of the present application;

[0045] FIG3( c ) is a method flow chart of a material identification method provided in an embodiment of the present application;

[0046] Figure 4(a) to Figure 4(f) A method flow chart of a material identification method provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of an application scenario of material identification by a material identification system provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of an application scenario of material identification by a material identification system provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of an application scenario of material identification by a material identification system provided in an embodiment of the present application;

[0050] Figure 8 A method flow chart of a material identification method provided in an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of the hardware results of a millimeter wave sensor provided in an embodiment of the present application;

[0052] Fig.10 A schematic diagram of the hardware results of an image sensor provided in an embodiment of the present application;

[0053] Fig.11 A schematic diagram of the hardware structure of a mobile electronic device provided in an embodiment of the present application;

[0054] Fig.12 A schematic diagram of the hardware structure of a control terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objectives, embodiments and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0056] refer to Figure 2 , Figure 2 The following is a schematic diagram showing an application scenario of material recognition performed by a material recognition system. Figure 2 As shown, the material recognition system 200 includes: a millimeter wave sensor 201, a control processing device 202, a bracket 203 and an object to be identified 204, wherein the millimeter wave sensor 201 and the object to be identified 204 are fixed on the bracket 203, and there is a preset distance between the millimeter wave sensor 201 and the object to be identified 204.

[0057] When performing identification, the millimeter wave sensor 201 sends millimeter waves (measurement signals) to the surface to be identified of the object to be identified 204, and receives multiple echo data returned by the surface to be identified of the object to be identified 204 at a detection time point or within a detection time period, wherein the echo data may include received power or electromagnetic scattering coefficient. The millimeter wave sensor 201 may send the echo data to the control processing device 202. The control processing device 202 may take the average value, median value, variance value or mean square error value of the multiple echo data as the overall echo data of the surface to be identified of the object to be identified 204. Since the corresponding echo data of different materials are different, the control processing device 202 may input the echo data into a pre-trained material recognition model to perform material recognition on the object to be identified 204.

[0058] but, Figure 2 The millimeter wave sensor 201 shown cannot detect the surface color and texture of the object to be identified, resulting in only a single material. Figure 2As shown, if the object to be identified 204 includes structure 2041 and structure 2042, wherein a surface of structure 2041 and a surface of structure 2042 are connected together to form a surface 2043 to be identified of the object to be identified 204, if the material, surface texture or color of structure 2041 and structure 2042 are different (for example, structure 2041 is made of concrete and structure 2042 is made of metal), the millimeter wave sensor 201 cannot determine the areas corresponding to the different materials in the surface to be identified of the object to be identified, that is, the areas corresponding to structures 2041 and 2042 respectively. Although there may be differences between the echo data of different materials received by the millimeter wave sensor 201, the control processing device 202 will still take the average value, median value, variance value or mean square error value, etc. in the echo data as the echo data of the entire surface to be identified of the object to be identified 204. Inputting the echo data into the material recognition model may only obtain the material recognition result of structure 2041 and structure 2042, or obtain the material recognition result of a material different from structure 2041 and structure 2042, resulting in a decrease in the accuracy of the recognition result obtained by the millimeter wave sensor 201 for material recognition.

[0059] In order to solve the problem of low accuracy of material recognition results in the prior art, the present application proposes a material recognition method, which adds an image sensor (such as a multispectral camera) to the material recognition system, captures an image of the surface to be identified of the object to be detected and identified through the image sensor, obtains spectral information of the surface to be identified of the object to be detected and identified based on the image, and obtains a first material recognition result corresponding to the surface to be identified and a material distribution result of the surface to be identified based on the spectral information. The material distribution result represents the distribution areas of different types of materials forming the surface to be identified (for example, the areas corresponding to the concrete material and the metal material mentioned above), that is, the material clustering result of the surface to be identified; in addition, the millimeter wave sensor can obtain the echo data of the surface to be identified, and fuse the material distribution result obtained above with the echo data, that is, superimpose the echo data on the material distribution result to determine the echo data corresponding to the distribution area of ​​each material in the surface to be identified; based on the echo data corresponding to the distribution area of ​​each material, a second material recognition result is obtained. It can be understood that the echo data here can be the echo data of the entire object to be identified or the surface to be identified of the object to be identified, which needs to be superimposed on the distribution area of ​​the object to be identified or the surface to be identified of the object to be identified so that each distribution area has corresponding echo data.

[0060] Then, the first material recognition result based on the spectral information and the second material recognition result based on the echo data can be fused and verified, for example, the material recognition result can be normalized, the material recognition result corresponding to the echo data can be calibrated with the material recognition result corresponding to the spectral information, etc., to obtain the final material recognition result. Based on the above scheme, accurate material recognition can be achieved.

[0061] In some embodiments, the material recognition result normalization here is used to limit the obtained material recognition result to a certain range, thereby eliminating the adverse effects caused by the data. For example, the first material recognition result may be 0.29, and the second material recognition result may be 0.34. Normalization may be to unify the material recognition results into a fixed range, for example: [0.30-0.40], that is, 0.29 may be normalized to 0.30.

[0062] In some embodiments, the method of calibrating the second material identification result through the first material identification result may include: adjusting the material identification result based on millimeter wave with the material identification result based on multi-channel spectral information, for example, taking the mean, variance value or mean square error value of the above two material identification results, etc.

[0063] For example, the above Figure 2 Taking the object 204 to be identified in the figure as an example, an image of the surface 2043 to be identified of the object 204 to be identified is captured by an image sensor, and the material identification results and material distribution results of the structures 2041 and 2042 corresponding to the surface 2043 to be identified are obtained; the millimeter wave sensor is controlled to obtain the echo data of the surface 2043 to be identified, and the echo data is superimposed on the material distribution result to obtain the echo data of the distribution area of ​​the structures 2041 and 2042 located in the surface 2043 to be identified, that is, the echo data of the respective areas of the structures 2041 and 2042; based on the echo data of the structures 2041 and 2042, the material identification results corresponding to the structures 2041 and 2042 are obtained. The material identification results based on the spectral information and the material identification results corresponding to the material clustering results obtained based on the echo data are normalized and calibrated to obtain the final material identification result of the surface 2043 to be identified of the object 204 to be identified.

[0064] It can also be seen that continuing to refer to Figure 2 , the millimeter wave sensor 201 in the material recognition system 200 needs to be set on a fixed and immovable detection platform (bracket 203), and the object to be recognized 204 also needs to remain stationary relative to the millimeter wave sensor. Figure 2 The material recognition system 200 is applied to various complex scenes in reality. The staff of the network operator needs to manually set the millimeter wave sensor 201 at multiple locations in the scene and frequently adjust the direction of the millimeter wave sensor 201 to complete the material recognition of the objects in the scene. This process requires a lot of detection costs.

[0065] In some embodiments, in order to solve the problem of high detection cost caused by the material recognition method corresponding to the fixed and immovable millimeter wave sensor, the image sensor and the millimeter wave sensor (recognition device) can also be fixedly set on the mobile electronic device. For example, a fixed base is set on the mobile electronic device, and the image sensor and the millimeter wave sensor are respectively installed on the fixed base, and the mobile electronic device is driven to move, such as: driving the mobile electronic device to move in multiple places in the scene to be detected to measure multiple objects to be identified. It can also ensure that the acquisition directions of the image sensor and the millimeter wave sensor are consistent, that is, the image sensor and the millimeter wave sensor face the same direction.

[0066] In some embodiments, if the area of ​​the surface to be identified of the object to be identified (structure to be identified) is larger than the single acquisition range of the image sensor and the millimeter wave sensor, the above-mentioned final material identification result may be the material identification result of the surface to be identified obtained at a measurement point during the movement of the mobile electronic device. The mobile electronic device may be driven to continue moving along the surface to be identified, and the material of the surface to be identified may be identified at multiple measurement points, and the material identification results obtained at multiple measurement points may be integrated into the final material identification result.

[0067] It can be seen that in the material recognition method of the embodiment of the present application, the material recognition of the object to be recognized is performed through multiple recognition devices, that is, the echo data obtained by the millimeter wave sensor is combined with the spectral information of the multi-spectral camera to further improve the accuracy of material recognition. The recognition device is set on a mobile electronic device, and the mobile electronic device is controlled to move in the environment to realize material recognition of multiple objects to be recognized in the environment or a larger surface to be recognized of the object to be recognized, thereby avoiding the artificial placement of recognition devices in the environment and reducing the detection cost.

[0068] The embodiments of the present application can be applied to material recognition in scenes / larger scenes, and then assist in building a propagation model based on ray tracing to achieve more accurate network planning, or assist in reshaping and controlling the electromagnetic response of the environment in communication perception integration, and other applications.

[0069] The sensor devices involved in the embodiments of the present application may include millimeter wave sensors, laser sensors, infrared sensors, image sensors, multi-spectral sensors, etc. The mobile electronic device may include an intelligent mobile robot, etc. It is understood that the embodiments of the present application do not impose any restrictions on the specific types of electronic devices. In order to more clearly and in detail introduce the material recognition method provided in the embodiments of the present application, the system architecture involved in implementing the method provided in the embodiments of the present application is first introduced below. Figure 3(a) and 3(b) The system architecture diagram for implementing the material identification method provided in the embodiment of the present application is exemplarily shown.

[0070] As shown in FIG. 3( a ), the system architecture 300 includes an image sensor 301 , a millimeter wave sensor 302 , a control module 303 , and a data analysis module 304 .

[0071] The image sensor 301 and the millimeter wave sensor 302 may also be referred to as sensor modules. The millimeter wave sensor 302 may be referred to as an active sensor, and the millimeter wave sensor 302 may transmit millimeter waves (measurement signals) and receive echo data of the measurement signals. The millimeter wave sensor 302 may transmit millimeter waves and receive echo data through a combination of multiple transmitting antennas and receiving antennas. The image sensor 301 may be a multispectral camera and may be referred to as a passive sensor, and the image sensor 301 may directly collect spectral information of the target scene.

[0072] The data analysis module 304 is used to obtain a material recognition result based on the echo data of the measurement signal and the material recognition result of the spectral information (multi-channel spectral information) and the material distribution result of the surface to be recognized.

[0073] The control module 303 is used to send control instructions to the image sensor 301 , the millimeter wave sensor 302 and the data analysis module 304 .

[0074] It can be seen that the system architecture 300 shown in FIG. 3( a ) can be configured in an electronic device, and the processor of the electronic device implements material recognition of the object to be recognized by running the control module 303 and the data analysis module 304 .

[0075] As shown in FIG. 3( b ), the system architecture 300 includes an image sensor 301 , a millimeter wave sensor 302 , a data analysis module 304 , a mobile electronic device 305 and a control terminal 306 .

[0076] The movable electronic device 305 can be called a motion device module or a motion device. The image sensor 301 and the millimeter wave sensor 302 can be fixedly set on the movable electronic device 305. The movable electronic device 305 can be moved to measure multiple objects to be identified or an object to be identified with a larger surface.

[0077] The control terminal 306 is used to send control instructions to the image sensor 301, the millimeter wave sensor 302, the data analysis module 304 and the mobile electronic device 305. The control terminal 306 can also be called a host computer, which is a computer device that sends control instructions. Usually, the host computer can provide a user operation interaction interface and display feedback results to the user.

[0078] Combination Figure 2 , Figure 3(a) and 3(b)The system architecture of the present invention is described below with reference to FIG3(c) as a flow chart of a material identification method according to an embodiment of the present application. The material identification method shown in FIG3(c) may include the following steps.

[0079] S301: Acquire target image data of an object to be identified.

[0080] For example, the object to be identified here can be Figure 2 The object to be identified 204 is shown as a target image data of a surface to be identified 2043 (the surface of the object to be identified 204) of the object to be identified 204, which is captured by an image sensor. The target image data here may be multi-channel spectral information.

[0081] S302: Divide the object to be identified into multiple material regions based on the target image data.

[0082] Exemplarily, based on the different manifestations of different colors or textures in the target image data of the to-be-identified surface 2043 of the to-be-identified object 204, for example, different colors or textures corresponding to different numerical changes in multi-channel spectral information, the material distribution results of the structures 2041 and 2042 corresponding to the to-be-identified surface 2043 are determined, that is, the material areas of the structures 2041 and 2042 in the to-be-identified surface 2043.

[0083] S303: Determine the millimeter wave sub-data of each material area from the millimeter data of the object to be identified.

[0084] Exemplarily, the millimeter data corresponding to the to-be-identified surface 2043 is obtained, and the millimeter wave sub-data corresponding to each material area is determined according to each material area.

[0085] S304: Obtaining a first material classification of each material region based on the millimeter wave sub-data of each material region.

[0086] Exemplarily, the millimeter-wave sub-data of each material region is matched with the millimeter-wave sub-data of a known material to obtain a material recognition result based on the millimeter-wave sub-data.

[0087] Combine the following Figure 4(a) to Figure 4(f) and Figure 5 Introducing the material recognition methods and scenarios mentioned in this application. Figure 5In the material recognition scenario shown, an image sensor and a millimeter wave sensor (not shown) can be fixedly installed on a mobile electronic device 505, and the control end 506 controls the mobile electronic device 505 to move to the object to be identified 501. The control end 506 controls the image sensor to capture an image of the object to be identified 501, and then obtains the material recognition results and material distribution results of the structures 5011 and 5012 corresponding to the object to be identified 501 based on the spectral information corresponding to the image; the millimeter wave sensor is controlled to obtain the echo data of the object to be identified 501, and the echo data is superimposed on the material distribution result to obtain the echo data of the distribution area of ​​the structures 5011 and 5012 located on the surface of the object to be identified 501, that is, the echo data of the respective areas of the structures 5011 and 5012; based on the echo data of the structures 5011 and 5012, the material recognition results corresponding to the structures 5011 and 5012 are obtained. The control end 506 normalizes and calibrates the material recognition result based on the spectral information and the material recognition result corresponding to each material clustering result obtained based on the echo data to obtain a final material recognition result of the object to be recognized 501 .

[0088] The material identification method is described in detail below with reference to FIG. 4( a ). The material identification method shown in FIG. 4( a ) may include the following steps.

[0089] S401: Controlling a multispectral camera to capture an optical image of a surface to be identified of an object to be identified, and obtaining multi-channel spectral information of the surface to be identified.

[0090] Exemplarily, the object to be identified may be a building partition in an office, and the millimeter wave sensor and the image sensor in the sensor module face the surface to be identified of the object to be identified. The control end 506 may also adjust the angle of the multi-spectral camera to obtain multi-channel spectral information of the surface to be identified of the object to be identified at multiple different angles. The multiple different angles may include a front view angle and multiple side view angles. For example: the front view angle may be the angle of the multi-spectral camera when the cross section of the multi-spectral camera is parallel to the surface to be identified of the object to be identified, that is, the multi-spectral camera is facing the surface to be identified of the object to be identified.

[0091] In some embodiments, the surface to be identified may also be referred to as a target imaging area, that is, the area of ​​the surface to be identified covered by the optical image, which may be the entire surface to be identified or a local area of ​​the surface to be identified.

[0092] S402: Based on the multi-channel spectral information, a material recognition result of the surface to be recognized is obtained, and the material recognition result is fused and overexposure compensated to obtain a final recognition result.

[0093] For example, step S402 here may include the steps shown in Fig. 4(b). By using the multi-channel spectral information at multiple angles to compensate each other, the overexposure problem of the optical image can be solved.

[0094] S402a: Making decisions on multi-channel spectral information at different angles through a spectral material decision model to obtain a material recognition result based on the multi-channel spectral information.

[0095] For example, the spectral material decision model here can be a pre-trained machine model. The spectral material decision model judges the multi-channel spectral information at different angles, and obtains the material recognition result of the surface to be recognized based on the multi-channel spectral information according to the amplitude information of the multi-channel spectral information under different bands of light and the trend information of the connecting lines of points in different bands. For example: the material recognition result can be Figure 5 The materials corresponding to the structures 5011 and 5012 in the object to be identified 501 are shown as follows: For example, the structure 5011 corresponds to wood, and the structure 5012 corresponds to concrete.

[0096] S402b: Fuse multi-channel spectral information at different angles and unify them into the same world coordinate system.

[0097] For example, since there may be errors between the multi-channel spectral information at different angles, the material recognition results may be inconsistent. It is necessary to unify the multi-channel spectral information of the object to be identified at multiple different angles into the same world coordinate system, that is, unify them into the same angle, for example: the coordinates of the multi-spectral camera at the normal viewing angle.

[0098] In some embodiments, an angle may be selected from different angles corresponding to the obtained multi-channel spectral information as a reference angle, for example, an orthoptic viewing angle is selected. Based on the angle difference between the orthoptic viewing angle and each angle, the multi-channel spectral information at each angle other than the orthoptic viewing angle is rotated or translated so that the multi-channel spectral information at different angles is unified to the orthoptic viewing angle.

[0099] S402c: Detecting an overexposed area corresponding to the optical image at the normal viewing angle, and obtaining a binary mask image corresponding to the overexposed area.

[0100] For example, the normal viewing angle here can be Figure 5The control terminal 506 shown in the figure pre-sets the angle of the multispectral camera. In the front view angle, that is, when the multispectral camera is facing the surface to be identified of the object to be identified, there may be an overexposed area in the optical image. The overexposed area here can be a highlight area that is overexposed in the optical image. A method for detecting the overexposed area can be to traverse the optical image through a preset detection window, the size of the preset detection window is smaller than the size of the optical image, and the mean of the pixel values ​​of each pixel point in the preset detection window is obtained. If the mean is outside the preset pixel value interval, for example: the mean is higher than the preset pixel value interval, then it is considered that the local area corresponding to the preset detection window is an overexposed area.

[0101] The binary mask image here can be a black and white image, that is, the overexposed area is binarized, which is to compare the grayscale value of the pixel in the overexposed area with the grayscale value threshold. The grayscale value higher than the grayscale value threshold is set to 255, otherwise it is set to 0. Each pixel has only two values, and the entire overexposed area appears in black and white.

[0102] S402d: Compare the binary mask image corresponding to the overexposed area with the optical image of the side view angle corresponding to the front view angle to obtain a correction area, and overlay the correction area on the optical image of the front view angle to obtain a corrected optical image of the overexposed area.

[0103] Exemplarily, using the binary mask image corresponding to the overexposed area to compare with the optical image of the side view angle corresponding to the front view angle is to determine the local area corresponding to the overexposed area of ​​the optical image of the front view angle in one or more optical images of the side view angle, that is, the correction area. The pixel values ​​of the pixels in the correction area are used to cover the overexposed area of ​​the optical image of the front view angle to obtain the optical image after the overexposed area is corrected.

[0104] It can be seen that steps S402b, S402c and S402d are not necessary, and there is no execution order between steps S402a to S402d. In other words, the optical image can be overexposed first, and then the material recognition result of the surface to be identified can be obtained based on the multi-channel spectral information, and the multi-channel spectral information at different angles can be fused and unified into the same world coordinate system. In some embodiments, the optical image can also be not overexposed.

[0105] S403: Based on the obtained multi-channel spectral information, a material clustering result of the to-be-identified surface of the to-be-identified object is obtained.

[0106] Exemplarily, the material clustering result here may be a material distribution result, which indicates the distribution area of ​​different types of materials forming the surface to be identified. For example, the distribution area of ​​different types of materials in the object to be identified may be determined based on the color and texture of different types of materials and the multi-channel spectral information. Figure 5 Taking the object 501 to be identified as an example, the multi-channel spectral information of the surface to be identified of the object 501 to be identified is obtained by the multi-spectral camera, and the material identification result corresponding to the surface to be identified is obtained, including two different materials. In other words, the surface to be identified may include distribution area 5011 and distribution area 5012, which may also be referred to as structure 5011 and structure 5012.

[0107] S404: Control the millimeter wave sensor to transmit millimeter waves to the object to be identified and collect echo data.

[0108] For example, the millimeter wave sensor here can also transmit millimeter waves to the surface to be identified of the object to be identified, and collect echo data. The echo data may include received power or electromagnetic scattering coefficient. It is understood that the millimeter wave sensor can transmit millimeter waves multiple times within a preset time period, and can also collect multiple echo data.

[0109] In some embodiments, when the multi-spectral camera captures an optical image of the surface to be identified of the object to be identified, the millimeter wave sensor can transmit millimeter waves multiple times through a combination of multiple transmitting antennas and receiving antennas, and collect multiple echo data. In other words, a set of multi-channel spectral information can correspond to multiple sets of echo data.

[0110] S405: Perform imaging processing on the echo data of the millimeter wave sensor to obtain an imaging result.

[0111] Imaging processing can be to convert echo data into an image composed of echo data, similar to the optical image captured by a multispectral camera. The echo data can be superimposed on the material distribution result to obtain the echo data corresponding to the distribution area of ​​different types of materials in the surface to be identified. It can be understood that the imaging result here can be the imaging result of the echo data of the entire object to be identified or the surface to be identified of the object to be identified, which needs to be superimposed on the material clustering result of the object to be identified or the surface to be identified of the object to be identified, that is, the distribution area of ​​the material of the object to be identified or the surface to be identified, so that each distribution area has a corresponding imaging result.

[0112] The imaging processing here may include: filtering the echo data through Fourier transform in frequency domain and time domain, projecting the echo data to each pixel point of the surface to be identified, and then delay compensating and coherently superimposing the echo data at each pixel point to obtain the imaging result of the echo data.

[0113] Exemplarily, step S405 here may include the steps shown in FIG. 4( c ).

[0114] S405a: Performing a range-direction Fourier transform on the echo data, so that the echo data is in the range frequency domain.

[0115] Exemplarily, the distance frequency domain here may be a frequency domain value of the echo data that can be obtained through Fourier transform, that is, an energy value of the echo data, that is, distance information.

[0116] S405b: Perform range-direction matched filtering processing on the echo data in the range frequency domain.

[0117] Exemplarily, the matched filtering process here is to remove larger or smaller values ​​from the frequency domain values ​​of the echo data, and calibrate the frequency domain values ​​of the echo data.

[0118] S405c: Perform inverse Fourier transform in the range direction on the echo data after matched filtering, so that the echo data is in the range time domain.

[0119] Exemplarily, the distance time domain here may be a time domain value of the echo data that can be obtained through Fourier transform, that is, the value change of the echo data over time, that is, time information.

[0120] S405d: Divide the surface to be identified into pixel areas, and sequentially calculate the time delays of all the transmitting antenna and receiving antenna combinations corresponding to the millimeter wave sensor relative to each pixel area.

[0121] Exemplarily, the surface to be identified here can also be referred to as a target imaging area, and the surface to be identified can correspond to multiple sets of echo data. The target imaging area can be the entire surface to be identified or a local area in the surface to be identified. Each pixel point or multiple pixel points in the surface to be identified can form a pixel point area, that is, a detection position on the object to be identified.

[0122] It is understandable that since the millimeter wave sensor can transmit millimeter waves multiple times within a preset time period, multiple echo data can be collected. The time delay here can be the time interval between the millimeter wave sensor transmitting millimeter waves multiple times.

[0123] S405e: Calculate the echo data of all the transmitting antenna and receiving antenna combinations corresponding to the millimeter wave sensor in sequence, traverse the pixel point area for coherent superposition, and obtain the imaging result of the surface to be identified.

[0124] Exemplarily, the echo data of each transmitting antenna and receiving antenna combination corresponding to the millimeter wave sensor is obtained and calculated in turn, each set of echo data is projected to each pixel area of ​​the surface to be identified, and then the echo data of each pixel area is coherently superimposed. The coherent superposition here is to perform phase compensation and delay compensation on the echo data at each moment, that is, the distance time domain, and then coherently superimpose to obtain the imaging result of the surface to be identified. It can be understood that the echo data of each transmitting antenna and receiving antenna combination can obtain an imaging result.

[0125] S406: Fusing the imaging result of the millimeter wave sensor and the material clustering result, and transmitting the fused material clustering result to the millimeter wave sensor.

[0126] For example, fusing the imaging result of the millimeter wave sensor and the material clustering result may be superimposing the imaging result corresponding to the echo data onto the material clustering result. Figure 6 As shown, through step S405, imaging results 1 to imaging results N can be obtained, and imaging results 1 to imaging results N can correspond to material clustering results 1 and material clustering results 2, that is, Figure 5 The distribution area 5011 and the distribution area 5012 in the surface to be identified of the object 501 to be identified are shown. The imaging results 1 to N are superimposed on Figure 5 The distribution area 5011 and the distribution area 5012 in the material clustering result of the object to be identified 501 shown can obtain the fused material clustering result, wherein the distribution area 5011 and the distribution area 5012 can correspond to echo data respectively.

[0127] It can be understood that the millimeter wave sensor can emit millimeter waves multiple times within a preset time period. For the material clustering results of the to-be-identified surface of the to-be-identified object obtained based on multi-channel spectral information, one material clustering result can also correspond to the imaging results of multiple echo data.

[0128] S407: Select imaging results corresponding to the same type of material from the fused material clustering results, and calculate a sequence of electromagnetic scattering coefficients of the material of the same type that varies with angle.

[0129] Exemplarily, step S407 here may include the steps shown in FIG. 4( d ).

[0130] S407a: Perform distance and antenna pattern compensation on the imaging result.

[0131] For example, continue to refer to Figure 6 , the imaging result here may be the material clustering result of the object to be identified, including the imaging results corresponding to material clustering result 1 and material clustering result 2. Since material clustering result 1 is obtained based on the multi-channel spectral information corresponding to the optical image, the distribution areas of material clustering result 1 and material clustering result 2 in the optical image can be determined.

[0132] The imaging result here may correspond to a combination of a transmitting antenna and a receiving antenna, and the imaging result, that is, the coherently superimposed echo data, may be compensated according to the antenna pattern and the distance. The distance here may be the distance between the transmitting antenna and the receiving antenna.

[0133] S407b: For imaging results corresponding to the same type of material, the actual incident angle corresponding to each imaging result is calculated using the three-dimensional information.

[0134] For example, one material clustering result may correspond to multiple imaging results. Accordingly, the same type of material in the material clustering result may also correspond to multiple imaging results. Figure 6 As shown, material clustering result 1 may correspond to imaging results 1 to imaging results N. The three-dimensional information here may be the coordinate information of the transmitting antenna and the receiving antenna. Since the millimeter wave sensor may transmit millimeter waves multiple times within a preset time period, the incident angle may be the angle at which the millimeter wave sensor transmits millimeter waves through the transmitting antenna each time, that is, the incident angle of the imaging result corresponding to each echo data.

[0135] S407c: Filter out the average value of the electromagnetic scattering coefficient corresponding to the imaging result of the actual incident angle corresponding to the preset incident angle.

[0136] Exemplarily, multiple preset incident angles may be pre-set, for example, the preset incident angles may include: 30°, 60°, 90°, etc., and the actual incident angle corresponding to the imaging result may be 30°, 45°, 55°, 60°, 90°, etc. Based on the preset incident angle, the electromagnetic scattering coefficients included in the imaging result that meets the preset incident angle in the actual incident angle are used to calculate the average value of the electromagnetic scattering coefficients of the multiple preset incident angles, that is, the average value of the electromagnetic scattering coefficients of the multiple imaging results corresponding to the same type of material in the material clustering result. It can be understood that the electromagnetic scattering coefficient here may be a data format of the echo data corresponding to the imaging result.

[0137] S407d: Based on the average values ​​of the multiple electromagnetic scattering coefficients, a sequence of electromagnetic scattering coefficients at multiple incident angles is obtained.

[0138] For example, for an imaging result in which a material clustering result may correspond to multiple echo data, average values ​​of multiple electromagnetic scattering coefficients may form an electromagnetic scattering coefficient sequence.

[0139] S408: Match the electromagnetic scattering coefficient sequence with the sample to obtain a material recognition result based on the echo data.

[0140] Exemplarily, by matching the electromagnetic scattering coefficient sequence that changes with the incident angle with the electromagnetic scattering coefficient sequence of a known material, a material recognition result based on the echo data can be obtained. In some embodiments, the matching method can include point matching of the sequence, or curve matching. Taking the point matching method as an example, the electromagnetic scattering coefficient sequence corresponding to multiple incident angles is obtained by actual measurement, and the electromagnetic scattering coefficient sequence is used to calculate the Euclidean distance between points with the electromagnetic scattering coefficient sequence of the corresponding angles of multiple materials in the sample library. The material corresponding to the electromagnetic scattering coefficient sequence with the smallest Euclidean distance in the sample library can be determined as the material recognition result.

[0141] In some embodiments, continue to refer to Figure 5 Based on the imaging results corresponding to the distribution areas 5011 and 5012 in the material clustering results of the object to be identified 501, the electromagnetic scattering coefficient sequences corresponding to the distribution areas 5011 and 5012 are obtained. By matching the two electromagnetic scattering coefficient sequences with the electromagnetic scattering coefficients of the corresponding angles of various materials in the sample library in a point matching manner, the material recognition results corresponding to the distribution areas 5011 and 5012 (for example, wood and concrete) can be obtained.

[0142] S409: Combining the material recognition results based on the multi-channel spectral information and the millimeter wave, outputting a fused material recognition result of the to-be-recognized surface of the to-be-recognized object.

[0143] For example, the material recognition result based on multi-channel spectral information and millimeter wave is to normalize and calibrate the material recognition result based on spectral information and the material recognition result corresponding to each material clustering result obtained based on echo data. Figure 5 The materials of the distribution area 5011 and the distribution area 5012 in the surface to be identified of the object to be identified 501 are shown (for example, wood and concrete).

[0144] In some embodiments, the material recognition results based on multi-channel spectral information and millimeter wave can be represented by numerical values. For example, taking distribution area 5011 as an example, the material recognition results based on millimeter wave can be 0.51 and 0.49, where 0.51 means 51% is metal and 0.49 means 49% is concrete. Relatively speaking, the material recognition results based on multi-channel spectral information can be 0.70 and 0.30. Calibration can include adjusting the material recognition results based on millimeter wave with the material recognition results based on multi-channel spectral information, for example, taking the mean, variance value or mean square error value of the above two material recognition results. For example, the mean of 0.51 and 0.70 can be 0.60, and the mean of 0.49 and 0.30 can be 0.39. It can be seen that by adjusting the material recognition results based on millimeter wave with the material recognition results based on multi-channel spectral information, the fused material recognition results of the object to be identified are more accurate.

[0145] In some embodiments, normalization may be to unify the material recognition results into a fixed range, for example: [0.30-0.40]. That is, 0.29 may be normalized to 0.30, indicating that the obtained material recognition results have a large deviation.

[0146] The following describes the method of matching the material recognition result according to the electromagnetic scattering coefficient sequence through Figure 4 (e) and Figure 4 (f). The point matching method may include the following steps.

[0147] S408a: Determine the electromagnetic scattering coefficient sequence.

[0148] For example, the electromagnetic scattering coefficient sequence here may be the electromagnetic scattering coefficient sequence of the object to be identified obtained in step S407, for example: Figure 5 The electromagnetic scattering coefficient sequences corresponding to the distribution area 5011 and the distribution area 5012 of the object 501 to be identified are shown.

[0149] S408b: Calculate the Euclidean distance between the electromagnetic scattering coefficient sequence and the electromagnetic scattering coefficient sequence of the samples in the sample library.

[0150] Exemplarily, the sample library here may be a preconfigured database, and the sample may be a sequence of electromagnetic scattering coefficients corresponding to a known material. For example, the Euclidean distance here may be a straight-line distance between the sample and each electromagnetic scattering coefficient in the sequence of electromagnetic scattering coefficients of the distribution area 5011 of the object to be identified 501. In some embodiments, the samples in the sample library may also include data on incident angles, and the incident angles of the electromagnetic scattering coefficient sequence and the incident angles of the samples may be matched simultaneously.

[0151] S408c: Determine the sample with the smallest Euclidean distance as the material recognition result.

[0152] Exemplarily, after calculating the Euclidean distance between the electromagnetic scattering coefficient sequence of the object to be identified obtained in step S407 and the electromagnetic scattering coefficient sequences corresponding to multiple materials in the sample library, the material with the smallest Euclidean distance is determined and the material is determined as the material identification result of the object to be identified.

[0153] In some embodiments, the curve matching method may include the following steps.

[0154] S408d: Determine the electromagnetic scattering coefficient sequence.

[0155] For example, the electromagnetic scattering coefficient sequence here may be the electromagnetic scattering coefficient sequence of the object to be identified obtained in step S407, for example: Figure 5 The electromagnetic scattering coefficient sequences corresponding to the distribution area 5011 and the distribution area 5012 of the object 501 to be identified are shown.

[0156] S408e: Use the interpolation function to interpolate the electromagnetic scattering coefficient sequence and fit the electromagnetic scattering coefficient curve.

[0157] For example, the interpolation function here is used to interpolate continuous values ​​between the electromagnetic scattering coefficients in the electromagnetic scattering coefficient sequence, for example: cubic spline interpolation, fitting the measured electromagnetic scattering coefficient sequence of the object to be identified to obtain the electromagnetic scattering coefficient curve

[0158] S408f: Calculate the structural similarity between the electromagnetic scattering coefficient curve and the sample curve using a structural similarity function.

[0159] Exemplarily, a structural similarity function (SSIM) is used to calculate the structural similarity between the measured electromagnetic scattering coefficient curve of the object to be identified and the sample curves corresponding to the various materials in the sample library. In some embodiments, the samples in the sample library may also include incident angle data, and the incident angle of the electromagnetic scattering coefficient sequence and the incident angle of the sample may be matched simultaneously.

[0160] S408g: Select the sample with the highest structural similarity as the material recognition result.

[0161] Exemplarily, after calculating the structural similarity between the measured electromagnetic scattering coefficient curve of the object to be identified and the curves corresponding to multiple materials in the sample library, the material corresponding to the curve with the highest similarity in the sample library is determined, and the material is determined as the material identification result of the object to be identified.

[0162] pass Figure 4(a) to Figure 4(f) as well as Figure 5 and Figure 6 A method flow chart of a material recognition method and a schematic diagram of a material recognition scene are described below. Figure 7 and Figure 8 A schematic diagram of another material identification scenario and another material identification method.

[0163] exist Figure 7 In the material recognition scenario shown, the millimeter wave sensor and the multispectral camera are configured on the motion device 705, and the control end 706 controls the motion device 705 to move along the surface to be recognized of the object to be recognized, and recognizes the surface to be recognized at multiple measurement points (measurement point 1 to measurement point N). The control end 706 can control and adjust the angles of the motion device 705, the millimeter wave sensor, and the multispectral camera. It can be seen that Figure 7 The material recognition scenario shown can be used for a larger object to be recognized or for performing material recognition on multiple objects to be recognized in an environment. Figure 8 A method flow chart of a material identification method in an embodiment of the present application is introduced, and the material identification method may include the following steps.

[0164] S801: command the motion device 705 to move to the measurement point 1, control the multi-spectral camera to capture the optical image of the surface to be identified of the object to be identified, and obtain the multi-channel spectral information of the surface to be identified.

[0165] For example, the control terminal 706 can adjust the angle of the motion device 705 or the multi-spectral camera to obtain multi-channel spectral information of the surface to be identified of the object to be identified at multiple different angles. The multiple different angles can include a front view angle and multiple side view angles.

[0166] S802: Based on the obtained multi-channel spectral information, a material recognition result of the surface to be recognized based on the multi-channel spectral information is obtained, and material recognition result fusion and overexposure compensation are performed.

[0167] Exemplarily, step S802 here is similar to step S402 in FIG. 4 , and the overexposure problem of the optical image can be solved by using the mutual compensation between the multi-channel spectral information at multiple angles.

[0168] S803: Obtaining a material clustering result of a to-be-identified surface of the to-be-identified object based on the obtained multi-channel spectral information.

[0169] Exemplarily, step S803 here is similar to step S403 of Fig. 4. The material clustering result here may be a material distribution result, and the material clustering result represents the distribution area of ​​different types of materials forming the surface to be identified.

[0170] S804: Control the millimeter wave sensor to transmit millimeter waves to the object to be identified and collect echo data.

[0171] Exemplarily, step S804 here is similar to step S404 of Figure 4. The millimeter wave sensor here can also be facing the surface to be identified of the object to be identified, transmit millimeter waves to the surface to be identified, and collect echo data. The echo data may include received power or electromagnetic scattering coefficient. It can be understood that the millimeter wave sensor can transmit millimeter waves multiple times within a preset time period, and can also collect multiple echo data. In some embodiments, when the multi-spectral camera captures the optical image of the surface to be identified of the object to be identified, the millimeter wave sensor can transmit millimeter waves multiple times through a combination of multiple transmitting antennas and receiving antennas, and collect multiple echo data. In other words, a set of multi-channel spectral information can correspond to multiple sets of echo data.

[0172] S805: Perform imaging processing on the echo data of the millimeter wave sensor to obtain an imaging result.

[0173] Exemplarily, step S805 here is similar to step S405 of FIG. 4 . The imaging processing here can be to convert the echo data into an image composed of echo data, similar to the optical image captured by a multi-spectral camera. The echo data can be superimposed on the material distribution result to obtain the echo data corresponding to the distribution area of ​​different types of materials in the surface to be identified. The imaging processing here may include: filtering the echo data through Fourier transform frequency domain and time domain, projecting the echo data to each pixel point of the surface to be identified, and then delay compensating and coherently superimposing the echo data at each pixel point to obtain the imaging result of the echo data.

[0174] S806: Fusing the imaging result of the millimeter wave sensor and the material clustering result, and transmitting the fused material clustering result to the millimeter wave sensor.

[0175] Exemplarily, step S806 here is similar to step S406 of Figure 4. The imaging result corresponding to the echo data is superimposed on the material clustering result to obtain the echo data corresponding to the distribution area 7011, the distribution area 7012, etc. in the surface to be identified of the object to be identified 701, that is, the optical image and echo data corresponding to the surface to be identified after fusion.

[0176] It can be understood that the millimeter wave sensor can emit millimeter waves multiple times within a preset time period. For the material clustering results of the to-be-identified surface of the to-be-identified object obtained based on multi-channel spectral information, one material clustering result can also correspond to the imaging results of multiple echo data.

[0177] S807: Select imaging results corresponding to the same type of material from the fused material clustering results, and calculate a sequence of electromagnetic scattering coefficients of the material of this type that varies with angle.

[0178] Step S807 here is similar to step S407 in FIG. 4 .

[0179] S808: Match the electromagnetic scattering coefficient sequence with the sample to obtain a material recognition result based on the echo data.

[0180] Exemplarily, step S808 here is similar to step S408 of FIG. 4 . According to the electromagnetic scattering coefficient sequence that changes with the incident angle, the material recognition result based on the echo data can be obtained by matching. In some embodiments, the matching method may include point matching of the sequence, or curve matching. Taking the point matching method as an example, the electromagnetic scattering coefficient sequence corresponding to multiple incident angles is obtained by actual measurement, and the electromagnetic scattering coefficient sequence is used to calculate the Euclidean distance between points with the electromagnetic scattering coefficients of the corresponding angles of multiple materials in the sample library. The material corresponding to the electromagnetic scattering coefficient with the smallest Euclidean distance in the sample library can be determined as the material recognition result.

[0181] S809: Combining the material recognition results based on the multi-channel spectral information and the millimeter wave, outputting a fused material recognition result of the to-be-recognized surface of the to-be-recognized object.

[0182] Exemplarily, step S809 here is similar to step S409 of Figure 4. The joint material recognition results based on multi-channel spectral information and millimeter wave are normalized and calibrated for the material recognition results based on spectral information and the material recognition results corresponding to the material clustering results obtained based on echo data.

[0183] In some embodiments, the control end 706 can control the motion device 705 to move to the measurement point 2, repeat the process from step S801 to step S809 until the measurement point N, complete the material recognition of the object to be recognized in the scene, and obtain the material recognition result of the scene.

[0184] The following will be combined Figures 9 to 12 , the hardware structures of the millimeter wave sensor 900, the image sensor 1000, the mobile electronic device 1100 and the control terminal 1200 to which the technical solution of the present application is applicable are introduced.

[0185] like Fig. 9 As shown, the millimeter wave sensor 900 may include a processor 910, an internal memory 920, a power management module 930, a battery 931, a wireless communication module 940, a detection module 950, and the like.

[0186] The processor 910 may include one or more processing units, for example, a central processing unit (CPU), a graphics processing unit (GPU), and the like.

[0187] The internal memory 920 may be used to store data, software programs, and modules. The internal memory 920 may include a program storage area (not shown) and a data storage area (not shown). The program storage area may store program code, which is used to enable the processor 910 to execute the material identification method provided in the embodiment of the present application by executing the program code.

[0188] The power management module 930 may include a power source, a power management component, etc. The power source may be a battery 931. The power management component is used to manage the charging of the power source and the power supply of the power source to other modules. The power management module 930 is used to connect the power source to the processor 910.

[0189] The wireless communication module 940 may include an antenna to provide a wireless local area network (WLAN) applied to the image sensor 1000. The millimeter wave sensor 900 may also communicate with the network and other devices through wireless communication technology. For example, the processor 910 may receive control instructions and the like through the wireless communication module 940.

[0190] The detection module 950 may include a receiving antenna and a transmitting antenna, and may generate millimeter waves and transmit and receive the millimeter waves through the receiving antenna and the transmitting antenna.

[0191] In some embodiments, the millimeter wave sensor 900 further includes buttons and indicators, etc. The buttons may include a power on / off button, etc. The indicators may include a laser indicator, a radio frequency indicator, an LED indicator, etc.

[0192] like Fig.10 As shown, the image sensor 1000 may include a processor 1010 , an internal memory 1020 , a power management module 1030 , a battery 1031 , a wireless communication module 1040 , a detection module 1050 , and the like.

[0193] The processor 1010 may include one or more processing units, for example, a central processing unit (CPU), a graphics processing unit (GPU), and the like.

[0194] The internal memory 1020 can be used to store data, software programs, and modules. The internal memory 1020 may include a program storage area (not shown) and a data storage area (not shown). The program storage area may store program code, which is used to enable the processor 1010 to execute the material identification method provided in the embodiment of the present application by executing the program code.

[0195] The power management module 1030 may include a power source, a power management component, etc. The power source may be a battery 1031. The power management component is used to manage the charging of the power source and the power supply of the power source to other modules. The power management module 1030 is used to connect the power source to the processor 1010.

[0196] The wireless communication module 1040 may include an antenna and may provide a wireless local area network (WLAN) applied to the image sensor 1000. The image sensor 1000 may communicate with the network and other devices through wireless communication technology. For example, the processor 1010 may receive control instructions and the like through the wireless communication module 1040.

[0197] The detection module 1050 may include a multi-channel spectral sensing module for collecting optical images.

[0198] In some embodiments, the image sensor 1000 further includes buttons and indicators, etc. The buttons may include a power on / off button, etc. The indicators may include a laser indicator, a radio frequency indicator, an LED indicator, etc.

[0199] The following will be combined Fig.11 , the hardware structure of the mobile electronic device (sports device) to which the technical solution of the present application is applicable is introduced. Figure 5 As shown, the mobile electronic device 1100 may include a processor 1110, an internal memory 1120, a drive motor 1130, a power management module 1140, a battery 1141, a wireless communication module 1150, a sensor module 1160, and the like.

[0200] The processor 1110 may include one or more processing units, for example, a central processing unit (CPU), a graphics processing unit (GPU), etc.

[0201] The internal memory 1120 can be used to store data, software programs and modules, and can be a volatile memory (Volatile Memory), such as a random access memory (Random-Access Memory, RAM). Specifically, the internal memory 1120 may include a program storage area (not shown) and a data storage area (not shown). The program storage area may store program code, which is used to enable the processor 1110 to control the movement of the mobile electronic device 1100 by executing the program code.

[0202] The driving motor 1130 is used to rotate to generate mechanical energy to drive the actuator of the mobile electronic device 1100 to move. By adjusting the rotation speed, the mobile electronic device 1100 can also complete a series of manually operated movements.

[0203] The power management module 1140 may include a power source, a power management component, etc. The power source may be a battery 541. The power management component is used to manage the charging of the power source and the power supply of the power source to other modules. The power management module 540 is used to connect the power source to the processor 510.

[0204] The wireless communication module 1150 may include an antenna, and provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), near field communication (NFC), infrared (IR), ZigBee, etc., applied to the mobile electronic device 1100. The mobile electronic device 1100 can communicate with the network and other devices through wireless communication technology.

[0205] The sensor module 1160 may include a proximity gyro sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a laser sensor, an odometer, and the like.

[0206] In some embodiments, the mobile electronic device 1100 further includes buttons and indicators, etc. The buttons may include a power on / off button, etc. The indicators may include a laser indicator, a radio frequency indicator, an LED indicator, etc.

[0207] The following will be combined Fig.12 , the hardware structure of the control terminal 1200 to which the technical solution of the present application is applicable is introduced. Among them, the control terminal 1200 can execute the material identification method provided by the present application.

[0208] like Fig.12 As shown, the control end 1200 may include a processor 1210, an internal memory 1220, a power management module 1230, a battery 1231, a wireless communication module 1240, and the like.

[0209] The processor 1210 may include one or more processing units, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microprocessor (MCU), an artificial intelligence (AI) processor, or a processing module or processing circuit of a programmable logic device (FPGA). Different processing units may be independent devices or integrated in one or more processors.

[0210] The internal memory 1220 can be used to store data, software programs and modules, and can be a volatile memory (Volatile Memory), such as a random access memory (Random-Access Memory, RAM); or a non-volatile memory (Non-Volatile Memory), such as a read-only memory (Read-Only Memory, ROM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, HDD) or a solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memory, or it can also be a removable storage medium, such as a secure digital (SecureDigital, SD) memory card. Specifically, the internal memory 1220 may include a program storage area (not shown) and a data storage area (not shown). The program storage area can store program code, and the program code is used to enable the processor 1210 to execute the material identification method provided in the embodiment of the present application by executing the program code.

[0211] The power management module 1230 may include a power source, a power management component, etc. The power source may be a battery 1231. The power management component is used to manage the charging of the power source and the power supply of the power source to other modules. The power management module 1230 is used to connect the power source to the processor 1210.

[0212] The wireless communication module 1240 may include an antenna, and transmit and receive electromagnetic waves via the antenna. The wireless communication module 1240 may provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), near field communication technology (NFC), infrared technology (IR), ZigBee, etc., applied to the control terminal 1200. The control terminal 1200 may communicate with the network and other devices through wireless communication technology.

[0213] For the control end 1200, the wireless communication module 1240 may also be replaced by a wired communication module such as a programmable logic controller (PLC).

[0214] Understandably, Fig. 9 , Fig.10 , Fig.11 and Fig.12 The structure shown does not constitute a specific limitation on the millimeter wave sensor 900, the image sensor 1000, the mobile electronic device 1100 and the control terminal 1200. In other embodiments of the present application, the millimeter wave sensor 900, the image sensor 1000, the mobile electronic device 1100 and the control terminal 1200 may include Fig. 9 , Fig.10 , Fig.11 and Fig.12 More or fewer components may be shown, or some components may be combined or separated, or the components may be arranged differently. Fig. 9 , Fig.10 , Fig.11 and Fig.12 The components shown may be implemented in hardware, software or a combination of software and hardware.

[0215] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0216] It should be understood that although the terms "first", "second", etc. may be used herein to describe various features, these features should not be limited by these terms. These terms are used only to distinguish and should not be understood as indicating or implying relative importance. For example, a first feature may be referred to as a second feature, and similarly a second feature may be referred to as a first feature without departing from the scope of the exemplary embodiments.

[0217] In addition, various operations will be described as multiple separate operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as implying that these operations must rely on the order of description, and many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can also be rearranged. When the described operations are completed, the process can be terminated, but there can also be additional operations not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0218] References in the specification to "one embodiment," "an embodiment," "an illustrative embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or property, but each embodiment may or may not necessarily include the particular feature, structure, or property. Moreover, these phrases are not necessarily directed to the same embodiment. In addition, when particular features are described in conjunction with a specific embodiment, the knowledge of those skilled in the art can affect the combination of these features with other embodiments, whether or not these embodiments are explicitly described.

[0219] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A / B" means "A or B." The phrase "A and / or B" means "(A), (B), or (A and B)."

[0220] As used herein, the term "module" may refer to, be part of, or include: a memory (shared, dedicated, or group) for running one or more software or firmware programs, an application-specific integrated circuit (ASIC), an electronic circuit and / or processor (shared, dedicated, or group), a combinational logic circuit, and / or other suitable components that provide the described functionality.

[0221] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not required. Instead, in some embodiments, these features may be described in a manner and / or order different from that shown in the illustrative drawings. In addition, the structural or method features included in a specific figure do not mean that all embodiments need to include such features. In some embodiments, these features may not be included, or these features may be combined with other features.

[0222] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the use of the technical solution of the present application is not limited to the various applications mentioned in the embodiments of the present patent. Various structures and variations can be easily implemented with reference to the technical solution of the present application to achieve the various beneficial effects mentioned herein. Various changes made within the knowledge of ordinary technicians in the field without departing from the purpose of the present application should all fall within the scope of the patent application.

Claims

1. A method for identifying the material of an object surface, applied to electronic equipment, characterized in that: include: Acquire first image data of an object to be identified; Dividing the object to be identified into a plurality of material regions based on the first image data; Determining the millimeter wave data of each material area from the millimeter wave data of the object to be identified; Based on the millimeter wave data of each of the material regions, a first wave material type of each material region is obtained.

2. The method according to claim 1, characterized in that The plurality of material regions include a first material region and a second material region, wherein the first material region and the second material region have different colors or textures.

3. The method according to claim 2, characterized in that Determining the millimeter wave data of each material area from the millimeter wave data of the object to be identified includes: Based on the millimeter wave imaging algorithm, obtaining at least one of time information, distance information and angle information included in the millimeter wave data of the object to be identified; Based on at least one of the time information, the distance information, and the angle information, confirming a detection position on the detected object to be identified corresponding to each millimeter wave data; The millimeter wave data belonging to each material area are divided based on the detection position of each millimeter wave data.

4. The method according to claim 3, characterized in that The obtaining of the first wave material type of each material area based on the millimeter wave data of each material area includes: The millimeter wave data of the first material region in each of the material regions is matched with the millimeter wave data of the first material sample, and correspondingly, when the matching result satisfies the matching condition, it is determined that the first material region has the first wave material type of the first material sample.

5. The method according to claim 4, characterized in that The obtaining of the first wave material type of each material region based on the millimeter wave data of each material region further includes: Determining that the first material area has a first image material type based on first image sub-data belonging to the first material area in the first image data; Based on the first wave texture type and the first image texture type, a texture type of the first texture area is determined.

6. The method according to claim 5, characterized in that The determining the material type of the first material area based on the first wave material type and the first image material type further includes: If the first wave material type probability corresponding to the first wave material type is lower than the first image material type probability corresponding to the first image material type, and the difference between the first wave material type probability and the first image material type probability exceeds a preset probability threshold, the first image material type probability is used to calibrate the first wave material type probability, and the material type of the first material area is determined based on the calibrated first wave material type probability.

7. The method according to claim 6, characterized in that The using the first image material type probability to calibrate the first wave material type probability comprises: The first image material type probability and the first wave material type probability are subjected to at least one of a mean, a variance, or a mean square error to obtain a calibrated first wave material type probability.

8. The method according to claim 5, characterized in that The determining the material type of the first material area based on the first wave material type and the first image material type further includes: If the first-wave material type probability corresponding to the first-wave material type is higher than the first-image material type probability corresponding to the first image material type, the material type of the first material area is determined according to the first-wave material type probability.

9. The method according to claim 1, characterized in that: The step of acquiring first image data of the object to be identified includes: Acquire multiple image data of the object to be identified taken from different angles; Selecting the first image data corresponding to the target angle from the plurality of image data; Corresponding to the overexposure phenomenon of the first image data, overexposure compensation is performed on the first image data based on the second image data among the multiple image data.

10. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device, and The processor is one of the processors of the electronic device, and is used to execute the material identification method described in any one of claims 1-9.

11. A computer program product, characterized in that The computer program product comprises instructions, which are used to implement the material identification method according to any one of claims 1 to 9 when executed by one or more processors.

Images