Image analysis method, electronic device, computer-readable storage medium, and camera
By using the gimbal device to move in a multispectral camera to obtain the initial and reference spectral images and calculate the spectral analysis results, the existing multispectral camera equipment has been solved, and efficient spectral analysis has been achieved. Reduced resource consumption.
Patent Information
- Application Number
- CN202410996959.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-07-24
AI Technical Summary
Due to the large size, high cost and poor reusability of existing multispectral camera equipment, the resource consumption for obtaining spectral analysis results is large, making it difficult to meet the needs of agricultural and other industries for efficient identification and advance intervention.
Using the image analysis method, the multi-spectral device is pre-divided into multiple spectral channels, and the gimbal device is used to drive the camera structure to obtain the initial and reference spectral images. Based on these images, the multi-spectral in the time-domain exchange airspace is realized. Image data acquisition to reduce resource consumption.
Without changing the camera volume, the reusability of the equipment is improved, the resource consumption for acquiring spectral analysis results is reduced, and the cost of use is reduced.
Smart Images

Figure CN118521654B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of cameras, and in particular to an image analysis method, an electronic device, a computer-readable storage medium and a camera. Background Art
[0002] With the development of intelligence, more and more industries have put forward higher requirements for camera technology. In the fields of agriculture, breeding and other industries, multispectral solutions are usually used to collect target images and analyze the images to achieve identification of target growth and early intervention. However, due to the long growth cycle of the target, the current use of multispectral solutions to collect target images often makes the camera equipment larger, the cost of use is high, the reusability is poor, and the resource consumption of obtaining spectral analysis results is large. Therefore, how to reduce the resource consumption of obtaining spectral analysis results has become an urgent problem to be solved. Summary of the invention
[0003] The main technical problem solved by the present application is to provide an image analysis method, an electronic device, a computer-readable storage medium and a camera, which can reduce the resource consumption for obtaining spectral analysis results.
[0004] In order to solve the above-mentioned technical problems, the first aspect of the present application provides an image analysis method, which is applied to a camera including a multi-spectral device and a pan-tilt device, and the multi-spectral device is pre-divided into multiple spectral channels. The method includes: obtaining initial spectral images when multiple observation sub-areas in an observation area correspond to their respective spectral channels in an initial state; wherein the number of the observation sub-areas matches the number of the spectral channels; using the pan-tilt device to drive the movement of part of the structure on the camera, and respectively obtaining reference spectral images when each observation sub-area corresponds to other spectral channels; based on the initial spectral image and the reference spectral image corresponding to each observation sub-area, determining the spectral analysis result corresponding to the target in each observation sub-area.
[0005] To solve the above technical problem, the second aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the method described in the first aspect.
[0006] In order to solve the above technical problem, the third aspect of the present application provides a computer-readable storage medium on which program data is stored. When the program data is executed by a processor, the method described in the first aspect is implemented.
[0007] To solve the above technical problems, the fourth aspect of the present application provides a camera, including a lens, an image sensor, a multi-spectral device and a pan-tilt device, and an electronic device as described in the second aspect above, and the electronic device is coupled to the image sensor and the pan-tilt device.
[0008] The above scheme collects the initial spectral images of the respective spectral channels corresponding to the multiple observation sub-areas in the observation area in the initial state, and then uses the pan-tilt device to drive the movement of some structures on the camera to collect the reference spectral images of each observation sub-area corresponding to other spectral channels, and based on the initial spectral image and the reference spectral image corresponding to each observation sub-area, determines the spectral analysis results corresponding to the targets in each observation sub-area, so that without changing the volume of the existing camera, multi-spectral image data collection is realized by exchanging time domain for space domain, and finally the spectral analysis results corresponding to all targets are obtained, which can improve the reusability of the camera and reduce the resource consumption of obtaining spectral analysis results. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:
[0010] Figure 1 It is a schematic diagram of a flow chart of an implementation method of the image analysis method of the present application;
[0011] Figure 2 is an imaging schematic diagram of four imaging sub-areas when the camera of the present application is in an initial position;
[0012] Figure 3 It is a structural schematic diagram of an embodiment of the electronic device of the present application;
[0013] Figure 4 It is a schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present application;
[0014] Figure 5 It is a structural schematic diagram of an embodiment of a camera of the present application. DETAILED DESCRIPTION
[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different implementation methods can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0016] It should be noted that the observation area of the present application includes plants with a long growth period such as trees and crops. The image analysis method proposed in the present application is applied to these plant targets.
[0017] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of the image analysis method of the present application. The method is applied to a camera including a multi-spectral device and a pan / tilt device, and the multi-spectral device is pre-divided into multiple spectral channels. The image analysis method includes:
[0018] S101: Acquire initial spectral images when a plurality of observation sub-regions in an observation region correspond to respective spectral channels in an initial state; wherein the number of observation sub-regions matches the number of spectral channels.
[0019] Specifically, initial spectral images of respective spectral channels corresponding to a plurality of observation sub-regions within the observation region in the initial state are collected and stored and recorded.
[0020] In one application, the initial state matches the initial position of the camera. After the camera is moved to the initial position, an initial spectral image of the respective spectral channels corresponding to multiple observation sub-areas in the observation area under the corresponding initial state when the camera is at the initial position is obtained.
[0021] In another application, the current position of the camera after startup is used as the corresponding position in the initial state, and the initial spectral image of multiple observation sub-areas in the observation area corresponding to their respective spectral channels in the initial state when the camera is at the current position is obtained.
[0022] In some implementation scenarios, the observation region is evenly divided into multiple observation sub-regions based on the area of the observation region.
[0023] In some implementation scenarios, a plurality of observation sub-areas are divided based on the types of observed objects in the observation area.
[0024] Optionally, the camera further includes a lens and an image sensor, wherein the lens may be a fixed-focus lens or a zoom lens, and the image sensor is used to sense light in an observation area and generate a spectral image.
[0025] S102: Using the pan-tilt device to drive some structures on the camera to move, and respectively obtaining reference spectral images of each observation sub-region corresponding to other spectral channels.
[0026] Specifically, the pan-tilt device is used to drive some structures on the camera to move, so as to collect reference spectral images of each observation sub-region corresponding to other spectral channels, and store and record them.
[0027] In one application method, the pan-tilt device is first used to drive a partial structure on the camera to move to the initial position of the camera, and then the pan-tilt device is used to drive a partial structure on the camera to move from the initial position based on a preset acquisition sequence, thereby obtaining a reference spectral image of each observation sub-area corresponding to other spectral channels.
[0028] In another application, a pan-tilt device is used to drive a part of the structure on the camera so that the camera moves directly from the current position, thereby obtaining a reference spectral image of each observation sub-region corresponding to other spectral channels.
[0029] It should be noted that since there is no need for real-time observation of plants with a long growth cycle, such as trees and crops, multispectral image data acquisition is achieved by exchanging time domain for spatial domain without changing the size of the existing camera, thereby improving the reusability of the camera and reducing the cost of use.
[0030] S103: Determine a spectrum analysis result corresponding to the target in each observation sub-region based on the initial spectrum image and the reference spectrum image corresponding to each observation sub-region.
[0031] Specifically, the initial spectral image and the reference spectral image corresponding to each observation sub-region are calculated and compared to obtain difference data between the initial spectral image and the reference spectral image, and tracking analysis is performed based on the difference data to determine the spectral analysis result corresponding to the target in each observation sub-region.
[0032] In one application, image features are extracted from a single initial spectral image corresponding to each observation sub-region and a single reference spectral image corresponding to other spectral channels of each observation sub-region, and calculations and comparisons are performed to obtain spectral analysis results corresponding to the target in each observation sub-region.
[0033] In another application method, image feature extraction is performed on multiple initial spectral images corresponding to each observation sub-region, and the image features of all the extracted initial spectral images are fused to obtain initial image fusion features. Image feature extraction is performed on multiple reference spectral images corresponding to other spectral channels of each observation sub-region, and the image features of all the extracted reference spectral images are fused to obtain reference image fusion features. Calculation and comparison are performed to obtain the spectral analysis results corresponding to the targets in each observation sub-region.
[0034] The above scheme collects the initial spectral images of the respective spectral channels corresponding to the multiple observation sub-areas in the observation area in the initial state, and then uses the pan-tilt device to drive the movement of some structures on the camera to collect the reference spectral images of each observation sub-area corresponding to other spectral channels, and based on the initial spectral image and the reference spectral image corresponding to each observation sub-area, determines the spectral analysis results corresponding to the targets in each observation sub-area, so that without changing the volume of the existing camera, multi-spectral image data collection is realized by exchanging time domain for space domain, and finally the spectral analysis results corresponding to all targets are obtained, which can improve the reusability of the camera and reduce the resource consumption of obtaining spectral analysis results.
[0035] In one embodiment, the multi-spectral device is a spectrometer, which is divided into a plurality of spectrometer regions of the same size, and each spectrometer region corresponds to a respective spectral channel, and the observation sub-region is obtained based on the following steps: based on the division method of the spectrometer, the image plane is divided into imaging sub-regions corresponding to the plurality of spectrometer regions respectively; based on the imaging sub-region corresponding to each spectrometer region, the observation sub-region on the object plane that has a mapping relationship with the imaging sub-region is determined.
[0036] Specifically, the beam splitter is pre-divided into a plurality of beam splitting regions of the same size, and each beam splitting region corresponds to a respective spectral channel, wherein each spectral channel is independent.
[0037] In one implementation scenario, the number of the light-splitting regions is at least three, and the single-side size of each light-splitting region is at least a preset pixel size, and each light-splitting region corresponds to its own coating and / or sub-wavelength structure.
[0038] Specifically, the number of divided light-splitting regions is at least three, and the single-side size of each light-splitting region is at least a preset pixel size. The spectral channel corresponding to each light-splitting region can be implemented by coating, sub-wavelength structure, etc.
[0039] In one specific implementation scenario, the number of divided spectroscopic areas is four, and the single-side size of each spectroscopic area is at least four times the size of a single pixel. In other specific implementation scenarios, the number of divided spectroscopic areas and the single-side size of each spectroscopic area are set according to actual conditions, and this application does not impose any specific restrictions on this.
[0040] Furthermore, the image plane is divided into imaging sub-regions corresponding to a plurality of spectral regions according to the division method of the spectral sheet, and an observation sub-region on the object plane that has a mapping relationship with the imaging sub-region is determined according to the imaging sub-region corresponding to each spectral region, wherein all observation sub-regions constitute a complete observation region.
[0041] In a specific implementation scenario, the image plane is divided into imaging sub-areas corresponding to a plurality of light-split areas in combination with the installation position of the camera, including the camera's calibration height, calibration angle, horizontal rotation angle, focal length, and corner settings. The selection of the camera's installation position should take into account factors such as the monitored target, ambient light conditions, and privacy protection, and the optimal setting should be determined in combination with the actual application scenario, and this application does not impose specific restrictions on this.
[0042] It can be understood that the position during calibration is the observation sub-region on the object plane that has a mapping relationship with the imaging sub-region, which is the initial position in the following text.
[0043] In one implementation scenario, the initial state matches the initial position of the camera. When the camera is in the initial position, each observation sub-region is imaged to the imaging sub-region with a mapping relationship through the corresponding spectral channel.
[0044] Specifically, when the camera is started normally, the camera is adjusted to an initial position. At this time, each observation sub-region is imaged onto an imaging sub-region with a mapping relationship through a corresponding spectral channel.
[0045] In a specific implementation scenario, please refer to Figure 2 , Figure 2It is an imaging schematic diagram when the number of imaging sub-areas is four when the camera of the present application is in the initial position. The image plane is divided into four imaging sub-areas, namely A', B', C' and D', each imaging sub-area has the same size, the spectral channel corresponding to imaging sub-area A' is λ1, the spectral channel corresponding to imaging sub-area B' is λ2, the spectral channel corresponding to imaging sub-area C' is λ3, and the spectral channel corresponding to imaging sub-area D' is λ4, wherein λ refers to the wavelength of light corresponding to a specific spectral region or band. According to each imaging sub-area, observation sub-areas A, B, C and D that have a mapping relationship with the imaging sub-area on the object plane are determined, and each observation sub-area is imaged onto the imaging sub-area that has a mapping relationship through the corresponding spectral channel, that is, observation sub-area A is imaged onto imaging sub-area A' via spectral channel λ1, observation sub-area B is imaged onto imaging sub-area B' via spectral channel λ2, and so on.
[0046] In this implementation scenario, obtaining the initial spectral images of the multiple observation sub-areas in the observation area in the initial state corresponding to their respective spectral channels in step S101 specifically includes: using a pan-tilt device to drive part of the structure on the camera to move to the initial position, and obtaining the initial spectral images of each of the observation sub-areas collected at the initial position corresponding to their respective spectral channels.
[0047] Specifically, the pan-tilt device is used to drive a part of the structure on the camera to move, so that the camera moves to an initial position, thereby obtaining an initial spectral image of each observation sub-area collected by the camera at the initial position when corresponding to the respective spectral channels.
[0048] Please continue reading Figure 2 Taking the four imaging sub-areas as an example, when the pan / tilt device drives the camera to move to the initial position, the initial spectral image of the spectral channel λ1 corresponding to the observation sub-area A is collected. Similarly, the initial spectral images of the spectral channels λ2, λ3 and λ4 corresponding to the observation sub-areas B, C and D are collected and stored.
[0049] In one implementation scenario, step S102 specifically includes: using a pan-tilt device to drive some structures on the camera to move so that each observation sub-area is imaged to other imaging sub-areas outside the mapping relationship through other spectral channels, and a reference spectral image of each observation sub-area corresponding to other spectral channels is obtained.
[0050] Specifically, the pan-tilt device is used to drive some structures on the camera to move, so that each observation sub-area can be imaged to other imaging sub-areas outside the mapping relationship through other spectral channels, thereby obtaining all reference spectral images of each observation sub-area corresponding to other spectral channels, thereby improving the accuracy of subsequent image analysis using spectral images.
[0051] Please continue reading Figure 2 Taking four imaging sub-areas as an example, the pan-tilt device is used to drive some structures on the camera to move, so that the observation sub-area A can be imaged to the imaging sub-area B' through the spectral channel λ2, thereby obtaining the reference spectral image of the observation sub-area A corresponding to λ2. Similarly, the pan-tilt device is continued to be used to drive some structures on the camera to move, and the reference spectral images of the observation sub-area A corresponding to the spectral channels λ3 and λ4 can also be obtained, as well as other reference spectral images of the observation sub-areas B, C and D corresponding to other spectral channels, and stored and recorded, which will not be repeated here.
[0052] In one implementation scenario, the camera also includes an ambient light detection device, which is used to obtain the light intensity of the observation area. Step S103 specifically includes: based on the initial spectral image and the reference spectral image corresponding to each observation sub-area, and the light intensity of the observation area, determining the spectral analysis result corresponding to the target in each observation sub-area.
[0053] Specifically, based on the initial spectral image and the reference spectral image corresponding to each observation sub-region, combined with the light intensity data of the observation region acquired by the ambient light detection device, a more accurate spectral analysis result corresponding to the target in each observation sub-region is obtained.
[0054] It is understandable that the light sensed by the image sensor may include some noise light in addition to the light reflected from the observation area. These noise lights may cause the obtained spectral analysis results to be inaccurate. Therefore, the ambient light detection device can be used to obtain the light intensity data of the observation area, and the dark noise correction can be used to reduce or even eliminate the impact of these noise lights on the spectral analysis results, thereby improving the accuracy of the obtained spectral analysis results.
[0055] In a specific implementation scenario, all the acquired initial spectral images and reference spectral images are preprocessed, and the initial spectral images and reference spectral images are corrected in combination with the light intensity data of the observation area acquired by the ambient light detection device. The complex spectral information is converted into indicators that can reflect the growth status of crops, such as NDVI (Normalized Difference Vegetation Index), NDWI (Normalized Water Index), etc. through an algorithm. The changes in the growth status of crops are identified by observing the indicators, and combined with professional knowledge, specific information about crop growth is obtained, such as leaf area index, biomass estimation, etc., and then the growth status of crops in the observation area is obtained, including normal growth, pests and diseases, or malnutrition.
[0056] See also Figure 3 , Figure 3: is a structural diagram of an embodiment of an electronic device of the present application, the electronic device 30 comprises a memory 300 and a processor 302 coupled to each other, wherein the memory 300 stores program instructions (not shown), and the processor 302 is used to execute the program instructions to implement the image analysis method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiment, which will not be repeated here. In addition, the processor 302 can also be called a CPU (Center Processing Unit). The processor 302 may be an integrated circuit chip with signal processing capabilities. The processor 302 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 302 can be implemented by an integrated circuit chip.
[0057] See also Figure 4 , Figure 4 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, the image analysis method in any of the above-mentioned embodiments is implemented. For descriptions of related contents, please refer to the detailed description of the above-mentioned method embodiments, which will not be repeated here.
[0058] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present implementation scheme.
[0059] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0060] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.
[0061] See also Figure 5 , Figure 5 It is a structural schematic diagram of an embodiment of a camera of the present application. The camera 50 includes a lens 500, an image sensor 501, a multi-spectral device 502 and a pan-tilt device 503, as well as the electronic device 30 described above, and the electronic device 30 is coupled to the image sensor 501 and the pan-tilt device 503.
[0062] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. An image analysis method, characterized in that: Applied to a camera including a multi-spectral device and a pan / tilt device, wherein the multi-spectral device is pre-divided into a plurality of spectral channels, the method comprises: Acquire an initial spectral image when a plurality of observation sub-regions in the observation region correspond to respective spectral channels in an initial state; wherein the number of the observation sub-regions matches the number of the spectral channels; Using the pan-tilt device to drive a part of the structure on the camera to move, respectively obtaining reference spectral images of each observation sub-area corresponding to other spectral channels; Based on the initial spectral image and the reference spectral image corresponding to each of the observation sub-regions, the spectral analysis result corresponding to the target in each of the observation sub-regions is determined; wherein the multi-spectral device is a spectrometer, the spectrometer is divided into a plurality of spectrometer regions of the same size, and each of the spectrometer regions corresponds to a respective spectral channel, and the observation sub-regions are obtained based on the following steps: based on the division method of the spectrometer, the image plane is divided into imaging sub-regions corresponding to the plurality of spectrometer regions respectively; based on the imaging sub-region corresponding to each of the spectrometer regions, the observation sub-region on the object plane that has a mapping relationship with the imaging sub-region is determined.
2. The image analysis method according to claim 1, characterized in that: The initial state matches the initial position of the camera. When the camera is in the initial position, each observation sub-region is imaged to the imaging sub-region having the mapping relationship through the corresponding spectral channel.
3. The image analysis method according to claim 2, characterized in that: The obtaining of initial spectral images of the multiple observation sub-areas in the observation area corresponding to respective spectral channels in the initial state includes: The pan-tilt device is used to drive a part of the structure on the camera to move to the initial position, and an initial spectral image of each observation sub-region collected at the initial position corresponding to the respective spectral channel is obtained.
4. The image analysis method according to claim 1, characterized in that: The method of using the pan-tilt device to drive a part of the structure on the camera to move, and obtaining reference spectral images of each observation sub-area corresponding to other spectral channels, includes: The pan-tilt device is used to drive part of the structure on the camera to move, so that each of the observation sub-areas is imaged to other imaging sub-areas outside the mapping relationship through other spectral channels, and a reference spectral image is obtained when each of the observation sub-areas corresponds to other spectral channels.
5. The image analysis method according to claim 1, characterized in that: The number of the light-splitting regions is at least three, and the single-side size of each of the light-splitting regions is at least a preset pixel size, and each of the light-splitting regions corresponds to a respective coating and / or sub-wavelength structure.
6. The image analysis method according to claim 1, characterized in that: The camera further comprises an ambient light detection device, wherein the ambient light detection device is used to obtain the light intensity of the observation area; Determining the spectral analysis result corresponding to the target in each observation sub-region based on the initial spectral image and the reference spectral image corresponding to each observation sub-region includes: Based on the initial spectral image and the reference spectral image corresponding to each of the observation sub-regions, and the light intensity of the observation region, a spectral analysis result corresponding to the target in each of the observation sub-regions is determined.
7. An electronic device, characterized in that: The method comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the method according to any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that: Program data is stored thereon, and when the program data is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
9. A camera, characterized in that: The invention comprises a lens, an image sensor, a multi-spectral device and a pan-tilt device, and an electronic device as claimed in claim 7, wherein the electronic device is coupled to the image sensor and the pan-tilt device.
Citation Information
Patent Citations
Banknote quality detection method and system based on multispectral image, and medium
CN113240643A
Image shooting method and device, terminal equipment and storage medium
CN114500857A
Camera module and electronic equipment
CN115242949A
Multi-view-field multispectral detection method, system and equipment and digital photography tool
CN116989895A
Three-dimensional integrated multispectral imaging sensor
US20170018594A1
