Methods, devices, and electronic equipment for coordinating millimeter-wave debris flow radar data and optical imagery.

Abstract

This disclosure provides a method, apparatus, and electronic device for coordinating millimeter-wave debris flow radar data and optical images. The method includes: acquiring radar data and optical images; detecting a target region, including a debris flow region, based on the optical images; associating the detection results of the radar data and the optical images with the target region based on the location information of the radar and image acquisition equipment; and coordinating the display of the detection results of the radar data and the optical images. This disclosure provides a more intuitive understanding of the target situation detected by radar, making operation more convenient.

Description

Technical Field

[0001] This invention relates to the field of debris flow monitoring, and in particular to a method, apparatus, and electronic device for coordinating millimeter-wave debris flow radar data and optical images. Background Technology

[0002] Landslides, collapses, and debris flows have occurred frequently in recent years, posing a serious threat to people's lives and property and the safety of engineering construction. With the rapid development of geological hazard monitoring technology, making geological hazard monitoring systems more accurate and improving user monitoring has always been a focus of the industry. Currently, geological hazard monitoring systems increasingly rely on data analysis and processing to achieve hazard monitoring. However, this data processing task is extremely complex, especially with radar detection data, which is large in volume and changes rapidly, making it difficult for users to intuitively perceive and judge. On the other hand, while video image data collected by cameras allows users to intuitively perceive the data, it lacks more precise data support such as distance and coordinates. If radar detection data and video image data could be integrated, users and developers would be able to more intuitively understand the radar-detected target situation through the geological hazard monitoring system, making operation more convenient and monitoring more accurate. This would also allow developers to more effectively develop geological hazard monitoring systems.

[0003] Existing debris flow monitoring methods typically involve establishing rudimentary monitoring points in the field, relying primarily on manual methods to record the specific circumstances on-site. This approach is technologically limited, has low accuracy in early warning and forecasting, and is easily affected by environmental conditions, making monitoring and early warning extremely difficult during storms. Furthermore, radar data is large in volume and changes rapidly, making it difficult for users to intuitively perceive and judge the situation. While video image data collected by cameras provides a more direct visual experience, it lacks more precise data such as distance and coordinates. Summary of the Invention

[0004] In view of the above-mentioned technical problems existing in the prior art, this disclosure proposes a method, device and electronic equipment for coordinating millimeter-wave debris flow radar data and optical images, which solves the technical problem that it is difficult to synchronously display radar monitoring data and image data of debris flow in the prior art, enabling a more intuitive understanding of the target situation detected by radar, making operation more convenient and monitoring more accurate.

[0005] According to a first aspect of this disclosure, a method for coordinating millimeter-wave debris flow radar data and optical imagery is provided, comprising:

[0006] Acquire radar data and optical images;

[0007] The target area is detected based on the optical image, and the target area includes a debris flow area;

[0008] Based on the location information of radar and image acquisition equipment, the detection results of the radar data and the detection results of the optical image are associated with the target area;

[0009] The detection results of the radar data and the detection results of the optical images are displayed in a coordinated manner.

[0010] In some possible implementations, the detection of the target region based on the optical image includes:

[0011] Convert the optical image into a grayscale image;

[0012] The grayscale image is segmented to obtain the target region;

[0013] and / or

[0014] Using the boundary information of the target area, the central axis of the target area is determined.

[0015] In some possible implementations, the step of associating the detection results of the radar data and the detection results of the optical image with the target area based on the location information of the radar and image acquisition devices includes:

[0016] Acquire measurement data from the radar at the location of the image acquisition device;

[0017] Obtain the first pitch angle of the sampling point on the central axis of the target area relative to the ground;

[0018] Read the altitude and distance information of the target area acquired via radar;

[0019] The measurement data is updated using the first pitch angle, the altitude information, and the distance information.

[0020] In some possible implementations, obtaining the first pitch angle of the sampling point on the central axis of the target area relative to the ground includes:

[0021] Obtain the dimensional information of the optical image, wherein the dimensional information includes the number of rows and columns;

[0022] A first matrix corresponding to the dimensional information is created according to a preset strategy;

[0023] Based on the parameters of the image acquisition device and the first matrix, the first pitch angle of the sampling point on the central axis is obtained.

[0024] In some possible implementations, a first matrix corresponding to the dimensional information is created according to a preset strategy, including:

[0025] An initial value is set for the first row of the matrix, wherein pixel values ​​are the same in the same row and different in different rows;

[0026] A second matrix is ​​created by increasing the pixel value by a preset value for each new row.

[0027] The first matrix is ​​generated by performing a difference operation between each pixel value in the second matrix and half of the pixel value in the last row of the second matrix.

[0028] In some possible implementations, obtaining the first pitch angle of the sampling points on the central axis based on the parameters of the image acquisition device and the first matrix includes:

[0029] The pitch angle corresponding to each pixel of the optical image is obtained based on the longitudinal calibration parameters of the image acquisition device and the arctangent relationship of the first matrix;

[0030] Based on the position of the central axis in the optical image and the pitch angle, a first pitch angle corresponding to each pixel on the central axis is determined.

[0031] According to a second aspect of this disclosure, a device for coordinating debris flow radar data and optical imagery is provided, comprising:

[0032] Radar and image acquisition equipment are used to acquire radar data and optical images, respectively;

[0033] The detection module is used to detect a target area based on the optical image, the target area including a debris flow area;

[0034] The association module is used to associate the measurement data corresponding to the radar data and the detection results of the optical image with the target area based on the location information of the radar and image acquisition equipment.

[0035] The display module is used to collaboratively display the optical image and the measurement data.

[0036] According to a third aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the method described in any one of the first aspects.

[0037] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described in any one of the first aspects.

[0038] Based on the above embodiments of this disclosure, this disclosure can use radar and image acquisition equipment to collect radar data and optical images of the target area respectively, and use the positional relationship between radar and image acquisition equipment to correct radar data, and display radar data in optical images in a coordinated manner, so as to realize the synchronous display of radar data and image data, and has the characteristics of accurate monitoring and intuitive monitoring. Attached Figure Description

[0039] Figure 1 A flowchart illustrating a method for coordinating millimeter-wave debris flow radar data and optical imagery according to an embodiment of the present disclosure is shown.

[0040] Figure 2 This is a flowchart illustrating the association of measurement data corresponding to the radar data with the target area according to an embodiment of the present disclosure;

[0041] Figure 3 This invention illustrates a device for coordinating debris flow radar data and optical images according to an embodiment of the present disclosure.

[0042] Figure 4 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown;

[0043] Figure 5 A block diagram of another electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0044] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0045] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0046] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0047] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0048] This disclosure provides a method for coordinating millimeter-wave debris flow radar data and optical imagery. The method can be executed by any information processing device, such as a terminal device, server, or other processing device. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, the method can be implemented by a processor calling computer-readable instructions stored in memory.

[0049] Specifically, Figure 1 A flowchart illustrating a method for coordinating millimeter-wave debris flow radar data and optical imagery according to an embodiment of the present disclosure is provided. The method for coordinating millimeter-wave debris flow radar data and optical imagery includes:

[0050] S10: Acquire radar data and optical images;

[0051] S20: Detect the target area based on the optical image, the target area including the debris flow area;

[0052] S30: Based on the location information of the radar and image acquisition equipment, associate the detection results of the radar data and the detection results of the optical image with the target area;

[0053] S40: Collaboratively display the detection results of the radar data and the detection results of the optical images.

[0054] Based on the above configuration, this embodiment of the present disclosure can realize the simultaneous monitoring of radar data and image data when a geological disaster occurs, and can update the radar data to the image for display, which can accurately realize the monitoring of the target area and intuitively display the monitoring data.

[0055] In some possible implementations, radar data can be used to detect geological disaster areas, and image data of the disaster areas can be acquired through image acquisition devices. The radar can be a millimeter-wave radar or other radar equipment, and the image acquisition devices include cameras or other devices capable of acquiring images of geological disaster areas in real time.

[0056] Before performing image acquisition, the image acquisition equipment must first be calibrated to establish its positional correspondence in the world coordinate system. Then, the millimeter-wave radar and image acquisition equipment can be activated to acquire radar monitoring data and optical depth data, respectively; the optical depth data is the optical image.

[0057] When optical images are acquired, segmentation processing of the target region in the optical images is performed. The target region is a geological disaster area, such as a debris flow area according to the embodiments of this disclosure. Specifically, in some possible implementations, the region of interest (target region) in the monitoring area can be segmented using a region growing method, or the region splitting and merging method or the watershed method can be used for this segmentation processing.

[0058] Specifically, the detection of the target region based on the optical image includes:

[0059] The optical image is converted into a grayscale image, for example, RGB can be converted into a grayscale image.

[0060] The grayscale image is segmented to obtain the target region;

[0061] The segmentation process of the region growing method includes:

[0062] First, determine the seed point for growth, i.e., obtain the starting point for region growth. According to growth rules, such as the eight-neighborhood method, compare the gray values ​​of the seed point with those of its eight surrounding pixels, and set a threshold: the absolute difference between a given point and its eight surrounding pixels must be less than the threshold. Then, determine if there are any new points that meet the growth conditions. If not, the process ends. If there are, first determine if the 3x3 neighborhood of this point is within the image range, and then again determine if the eight surrounding pixels meet the threshold condition. Record the number of new growth points. Finally, display the image after region segmentation. The threshold can be a pre-set value, such as 0.5, or it can be adaptively adjusted using an adaptive threshold determination method, but this is not a specific limitation of this disclosure.

[0063] The segmentation process of the region splitting and merging method includes: first, arbitrarily dividing the input image into initial regions; then, setting similarity criteria for the regions based on the statistical characteristics of the image, splitting or merging these regions to meet the segmentation requirements.

[0064] The watershed segmentation process involves: based on three-dimensional visualization of the image, where two parameters are coordinates and the other is grayscale. According to topography, there are three types of points:

[0065] ① Local minimum point

[0066] ②When water is placed at a certain point, it will fall to a single, minimum point.

[0067] ③ When water is at a certain position, it will flow proportionally to more than one such minimum point.

[0068] The set of points that satisfy condition ② is called the catchment basin or watershed of this minimum value, and the set of points that satisfy condition ③ forms the peak line of the terrain surface.

[0069] When a target region is determined, embodiments of this disclosure can also utilize the boundary information of the target region to determine the central axis of the target region. The method for determining the central axis of the target region may include saving the boundary pixels of the segmented region; obtaining the pixel information of the central axis of the segmented region by adding the pixels on both sides and then dividing by 2; and finally saving the obtained pixel information, which includes the position information of the central axis.

[0070] In some possible implementations, in order to achieve coordination between optical imagery and radar data of the target area, it is also necessary to fuse radar data and optical imagery, and associate the detection results of radar data and optical imagery with the target area.

[0071] Figure 2 A flowchart illustrating the association of detection results from radar data and optical imagery with the target area according to an embodiment of this disclosure is provided. The step of associating the detection results from radar data and optical imagery with the target area based on the location information of the radar and image acquisition device includes:

[0072] S301: Obtain measurement data corresponding to the radar data at the location of the image acquisition device;

[0073] S302: Obtain the first pitch angle of the sampling point on the central axis of the target area relative to the ground;

[0074] S303: Reads altitude and distance information of ground target areas acquired via radar;

[0075] S304: Update the measurement data using the first pitch angle, the altitude information, and the distance information.

[0076] In some possible implementations, the radar can first be positioned at the location of the image acquisition device to obtain initial radar data. Measurement data from this radar data in the target area can then be read, including height information and distance information from the radar. Subsequently, during subsequent data acquisition, due to radar movement, the acquired radar data needs to be converted back to the initial state (camera position), i.e., the radar data needs to be transferred.

[0077] Specifically, steps S301 and S302 can be executed simultaneously or separately, and the execution order is not specifically limited. In some possible implementations, obtaining the first elevation angle of the sampling points on the central axis of the target area relative to the ground includes: obtaining the dimensional information of the optical image, the dimensional information including the number of rows and columns; creating a first matrix corresponding to the dimensional information according to a preset strategy; and obtaining the first elevation angle of the sampling points on the central axis according to the parameters of the image acquisition device and the first matrix. Furthermore, obtaining the first elevation angle of the sampling points on the central axis according to the parameters of the image acquisition device and the first matrix includes: obtaining the elevation angle corresponding to each pixel of the optical image according to the longitudinal calibration parameters of the image acquisition device and the arctangent relationship of the first matrix; and determining the first elevation angle corresponding to each pixel on the central axis based on the position of the central axis in the optical image and the elevation angle.

[0078] In one example, the pitch angle of each pixel in the acquired optical image can be calculated using image encoding. First, the size of the optical image, i.e., the number of rows and columns, is obtained. Then, a matrix is ​​created based on the number of rows and columns, where each row has the same pixel value and each row increments from 1 to the row number. Next, each number in the matrix is ​​subtracted by half the value of the last row. Then, based on the camera calibration parameters and the obtained matrix, the pitch angle corresponding to each pixel in the optical image is obtained through the arctangent relationship, i.e., the pitch angle = 2 * actan(0.5 * A / y), where A is the matrix created above, and y is the camera's longitudinal calibration parameter. Finally, by traversing the central axis, the pitch angle information corresponding to each pixel on the central axis is extracted and converted to angles, i.e., the obtained pitch angle is multiplied by 180 and then divided by π. At this point, assuming the camera is horizontally placed, the central optical axis emits horizontal light, and the pitch angle is the angle with the horizontal relative to the ground.

[0079] In some embodiments, this disclosure directly utilizes the altitude and distance information of the target area acquired via radar. However, during actual data acquisition, moving the radar position causes the altitude and distance information to change relative to a new coordinate system. Therefore, based on the positional relationship between the radar coordinates and the camera coordinates, as well as the acquired original altitude and distance information, a new altitude and distance can be calculated to correct the measurement data obtained by the radar. Therefore, embodiments of this disclosure can utilize the first elevation angle and the altitude and distance information to update the measurement data, including:

[0080] Based on the first measurement data of the target area monitored by the radar at the first moment, the first position association of the radar and the image acquisition device at the first moment, and the position association of the radar and the image acquisition device at the second moment, the second measurement data at the second moment is determined, wherein the first moment is earlier than the second moment. In this embodiment, new height information can be obtained based on the coordinate difference between the radar and the camera, and new distance information can be obtained based on the trigonometric function relationship between the height information and the distance information. The new relative height between each target point and the radar is denoted as h, and the new distance is denoted as R. Based on h and R, using the arctangent function atan, the angle between the line connecting the target point and the radar and the horizontal plane of the ground is calculated and denoted as α. Then, α is converted to an angle, i.e., the new elevation angle is multiplied by 180 and divided by π to obtain the second elevation angle. Then, based on the second elevation angle obtained by the radar and the first elevation angle of the central axis in the target area, the radar measurement data and the target area of ​​the optical image are matched and fused, that is, the measurement data corresponding to the matched first and second elevation angles are matched and coordinated with the sampling points on the central axis.

[0081] This disclosure uses the length of the sampling points on the central axis as the number of iterations, and calls a minimum function (where the difference between the two is minimized) based on the first and second pitch angles. The parameters are the pitch angle of the processed optical image and the absolute value of the new pitch angle, respectively. The return value is the row number of the point where fusion was successful. The corresponding sampling point is then found through the row number, thereby updating and matching the measurement data of the sampling points in each row. This disclosure can use points on the central axis corresponding to each row as sampling points, but this is not intended to limit the scope of this disclosure.

[0082] Having obtained measurement data from each sampling point along the central axis, the corresponding measurement data can be displayed on the optical image, thereby showing relevant data on geological hazards in the optical image in real time. Furthermore, it can update and display dynamic data changes in the video image in real time.

[0083] In addition, the optical images in this embodiment can be video data, so target area tracking processing can be performed on the optical images of each time frame in the video data to obtain the movement speed information of geological disasters.

[0084] In some implementations, video images captured by a high-speed camera can be read, with each frame read and the frame number saved. Then, a feature point matching method is used to detect the target. First, the initial first frame is read, and the target's feature points are obtained using methods such as SURF. Then, all frames except the first frame are traversed, and each subsequent frame is treated as the current frame. Feature points are obtained using the same method, and the `matchFeatures` function is called to match the feature points of the current frame with those of the first frame. A threshold is set based on the number of matching points. When the number of matching points is greater than the threshold, the target is considered successfully detected and its position is marked. If the number of matching points is less than the threshold, the matching is considered a failure. Then, the position is predicted using methods such as a Kalman filter. First, the velocity and position information of the successfully detected target are fed into the filter for real-time updates. Then, the `predict` function is called based on the target's previous motion state to predict its position, updating the filter's velocity and position information, and marking the predicted target position.

[0085] In addition, the embodiments of this disclosure can also utilize optical flow or background difference detection algorithms to track the target region, wherein optical flow can include:

[0086] At a specific moment of motion, the pixels on the image are projected onto the points on the three-dimensional object; then, the image is dynamically analyzed based on the velocity vector characteristics of each pixel. If there is no moving target in the image, the optical flow vector changes continuously throughout the entire image area. If the target moves relative to the image background, the velocity vector formed by the moving target will necessarily be different from the velocity vector of the neighboring background, thereby detecting the specific position and related motion parameters of the moving target.

[0087] Background difference detection algorithms may include: obtaining a background model through background modeling; after establishing the background model, performing difference between each frame of the acquired video sequence and the background image; performing binarization processing on the obtained difference image; when a pixel in the difference image is greater than a threshold, it is considered a moving target pixel, otherwise it is a background pixel.

[0088] Based on the above embodiments of this disclosure, this disclosure can use radar and image acquisition equipment to collect radar data and optical images of the target area respectively, and use the positional relationship between radar and image acquisition equipment to correct radar data, and display radar data in optical images in a coordinated manner, so as to realize the synchronous display of radar data and image data, and has the characteristics of accurate monitoring and intuitive monitoring.

[0089] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further.

[0090] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0091] In addition, this disclosure also provides a device, electronic equipment, computer-readable storage medium, and program for coordinating debris flow radar data and optical images. All of the above can be used to implement any of the methods provided in this disclosure. The descriptions of the corresponding technical solutions and the corresponding records in the method section will not be repeated here.

[0092] Figure 3 This invention discloses a device for coordinating debris flow radar data and optical imagery according to an embodiment of the present disclosure, comprising:

[0093] Radar 100 and image acquisition device 200 are used to acquire radar data and optical images, respectively;

[0094] Detection module 300 is used to detect a target area based on the optical image, the target area including a debris flow area;

[0095] The association module 400 is used to associate the measurement data corresponding to the radar data and the detection results of the optical image with the target area based on the location information of the radar and image acquisition equipment.

[0096] The display module 500 is used to collaboratively display the detection results of the radar data and the detection results of the optical images.

[0097] The detection of the target region based on the optical image includes:

[0098] Convert the optical image into a grayscale image;

[0099] The grayscale image is segmented to obtain the target region;

[0100] and / or

[0101] Using the boundary information of the target area, the central axis of the target area is determined.

[0102] In some possible implementations, the step of associating the detection results of the radar data and the detection results of the optical image with the target area based on the location information of the radar and image acquisition devices includes:

[0103] Acquire measurement data from the radar at the location of the image acquisition device;

[0104] Obtain the first pitch angle of the sampling point on the central axis of the target area relative to the ground;

[0105] Read the altitude and distance information of the target area acquired via radar;

[0106] The measurement data is updated using the first pitch angle, the altitude information, and the distance information.

[0107] In some possible implementations, obtaining the first pitch angle of the sampling point on the central axis of the target area relative to the ground includes:

[0108] Obtain the dimensional information of the optical image, wherein the dimensional information includes the number of rows and columns;

[0109] A first matrix corresponding to the dimensional information is created according to a preset strategy;

[0110] Based on the parameters of the image acquisition device and the first matrix, the first pitch angle of the sampling point on the central axis is obtained.

[0111] In some possible implementations, a first matrix corresponding to the dimensional information is created according to a preset strategy, including:

[0112] An initial value is set for the first row of the matrix, wherein pixel values ​​are the same in the same row and different in different rows;

[0113] A second matrix is ​​created by increasing the pixel value by a preset value for each new row.

[0114] The first matrix is ​​generated by performing a difference operation between each pixel value in the second matrix and half of the pixel value in the last row of the second matrix.

[0115] In some possible implementations, obtaining the first pitch angle of the sampling points on the central axis based on the parameters of the image acquisition device and the first matrix includes:

[0116] The pitch angle corresponding to each pixel of the optical image is obtained based on the longitudinal calibration parameters of the image acquisition device and the arctangent relationship of the first matrix;

[0117] Based on the position of the central axis in the optical image and the pitch angle, a first pitch angle corresponding to each pixel on the central axis is determined.

[0118] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0119] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0120] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0121] Figure 4 This diagram illustrates a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, or other terminal.

[0122] Reference Figure 4 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0123] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0124] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0125] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0126] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0127] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0128] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0129] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0130] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0131] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0132] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

[0133] Figure 5 A block diagram of another electronic device 1900 according to an embodiment of the present disclosure is shown. For example, electronic device 1900 may be provided as a server. (Refer to...) Figure 5 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0134] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0135] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0136] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0137] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0138] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0139] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0140] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0141] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0142] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0144] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for coordinating millimeter-wave debris flow radar data and optical imagery, characterized in that, include: Acquire radar data and optical images; The target area is detected based on the optical image, and the target area includes a debris flow area; Based on the location information of radar and image acquisition equipment, the detection results of the radar data and the detection results of the optical image are associated with the target area; The detection results of the radar data and the detection results of the optical images are displayed collaboratively. in: The method of associating the detection results of the radar data and the detection results of the optical image with the target area based on the location information of the radar and image acquisition equipment includes: Read the measurement data of the radar at the location of the image acquisition device; Obtain the first pitch angle of the sampling point on the central axis of the target area relative to the ground; Read the altitude and distance information of the target area obtained through radar; The measurement data is updated using the first pitch angle, the altitude information, and the distance information.

2. The method according to claim 1, characterized in that, The detection of the target region based on the optical image includes: Convert the optical image into a grayscale image; The grayscale image is segmented to obtain the target region; Using the boundary information of the target area, the central axis of the target area is determined.

3. The method according to claim 1, characterized in that, Obtaining the first pitch angle of the sampling point on the central axis of the target area relative to the ground includes: Obtain the dimensional information of the optical image, wherein the dimensional information includes the number of rows and columns; A first matrix corresponding to the dimensional information is created according to a preset strategy; Based on the parameters of the image acquisition device and the first matrix, the first pitch angle of the sampling point on the central axis is obtained.

4. The method according to claim 3, characterized in that, The first matrix corresponding to the dimensional information is created according to a preset strategy, including: An initial value is set for the first row of the matrix, wherein pixel values ​​are the same in the same row and different in different rows; A second matrix is ​​created by increasing the pixel value by a preset value for each new row. The first matrix is ​​generated by performing a difference operation between each pixel value in the second matrix and half of the pixel value in the last row of the second matrix.

5. The method according to claim 3 or 4, characterized in that, Based on the parameters of the image acquisition device and the first matrix, the first pitch angle of the sampling points on the central axis is obtained, including: The pitch angle corresponding to each pixel of the optical image is obtained based on the longitudinal calibration parameters of the image acquisition device and the arctangent relationship of the first matrix; Based on the position of the central axis in the optical image and the pitch angle, a first pitch angle corresponding to each pixel on the central axis is determined.

6. A device for coordinating debris flow radar data and optical images, characterized in that, include: Radar and image acquisition equipment are used to acquire radar data and optical images, respectively; The detection module is used to detect a target area based on the optical image, the target area including a debris flow area; The association module is used to associate the measurement data corresponding to the radar data and the detection results of the optical image with the target area based on the location information of the radar and image acquisition device according to claim 1. The display module is used to collaboratively display the optical image and the measurement data.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, The computer program instructions, when executed by a processor, implement the method described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method described in any one of claims 1-5.

Images