Image processing method for determining steering commands for control wings of a steering body

Description

[0001] The invention relates to an image processing method for determining steering commands for control wings of a guided missile.

[0002] Guided missiles equipped with an imaging system are designed to detect a target and autonomously steer towards it. To detect a distant target, high-resolution optics and powerful image processing are essential for rapidly processing the data from a high-resolution image sensor. Furthermore, such a guided missile must be agile to reliably track an evasive target. To achieve this speed, the image processing system must operate very quickly to minimize detection delays. These requirements place high demands on the image processing hardware, which must be as compact as possible to allow for a small missile design.

[0003] To achieve these goals, the hardware of a guided missile's image processing system, in conjunction with the software used, is highly optimized to create a compact system precisely tailored to the tasks at hand. The individual image processing steps—from object recognition in the image and determination of the object's coordinates to object classification—are sequentially optimized and precisely timed to ensure optimal utilization of the available hardware.

[0004] German patent DE 692 05 481 T discloses a method for the self-guided guidance of a missile against a target by means of distance measurements. For this purpose, the missile comprises a detection and tracking unit in which a common data bus connects memory, a control and processing unit, and various operators. Images supplied by an infrared detector are processed by the operators and the control and processing unit.

[0005] US Patent 5 341 142 A deals with an automatic target acquisition and tracking system for a focal plane array seeker. Automatic target acquisition is achieved through three independent target acquisition algorithms: maximum likelihood classification, spatial video clustering, and target-to-interference ratio. Each algorithm operates asynchronously and delivers independent target detection results. Target information is then hierarchically combined and prioritized probabilistically. The highest-priority target is passed to a dual-mode tracker consisting of a minimum absolute difference correlation tracker and a center of gravity tracker. The dual-mode tracker then provides a feedback signal to a proportional navigation system or other guidance / control system to direct the trajectory of a projectile.

[0006] It is an object of the present invention to provide an image processing method for determining steering commands for control wings of a guided missile, which is further improved with regard to its usability compared to known systems.

[0007] This problem is solved by an image processing method according to the features of claim 1, in which, among other things, a camera records an environment, generates an image data set from the recording and makes the image data set available on a data bus, several image processing units connected to the data bus process data available on the data bus and make their work results available on the data bus, a data fusion combines the work results assigned to the image data set into a main result and an autopilot calculates steering commands for the control wings from the main result.

[0008] By arranging the image processing units on a data bus, they can be flexibly adapted to the specific application of the guided missile by simply connecting and using the appropriate unit. Image processing can be easily specialized, allowing for optimized use of hardware resources for specific applications. Furthermore, this approach offers the advantage of longer processing times for each image processing unit compared to a fixed clock cycle, enabling the efficient handling of computationally intensive specialized tasks while minimizing hardware strain.

[0009] The guided missile is advantageously an unmanned aerial vehicle with a seeker head and a control system equipped for autonomous guidance of the missile based on a target guidance system. The guided missile includes, for example, a warhead for engaging a target by detonation. The camera is advantageously part of the seeker head of the guided missile and preferably includes optics for focusing ambient radiation onto a detector of the camera. Advantageously, the optics are infrared optics and the detector is an infrared detector.

[0010] From a single image, the camera generates an image data set. This can contain an image of the surroundings or other compiled data from which, for example, an image of the surroundings can be generated. The camera is connected to the data bus, to which the image processing units and the data fusion unit are also connected. The data fusion unit calculates the main result from the processing output of the image processing units.

[0011] The camera conveniently records the surroundings several times in succession.

[0012] Each image capture, or multiple captures, is used to create an image data set, which can then be made available sequentially on the data bus. These image data sets can be accessed and processed simultaneously or sequentially by one or more image processing units.

[0013] The image processing units process the data available on the data bus. This can be the image data set or a result derived from the image data set by one of the image processing units. Ideally, one image processing unit processes the image data set into a result, and a second image processing unit then takes this result and processes it into its own result. Optionally, one or more additional image processing units can take the most recent result and process it further. In this way, successive results for a single image data set can be available on the data bus, either sequentially or simultaneously. It is also possible for multiple image processing units to work on the same image data set or the same result and make different results available on the data bus.

[0014] Data fusion combines several work results into one main result.

[0015] The main result can contain the output of one or more image processing units, or only parts thereof. The main result is designed to serve as input data for the autopilot to calculate steering commands for the control surfaces. Using these commands, the missile can be guided to a target, advantageously autonomously. When multiple image datasets are involved, the data fusion process expediently combines outputs associated with a single image dataset into a main result and performs this process for multiple or all image datasets.

[0016] In an advantageous embodiment of the invention, one of the image processing units extracts at least one object from the image data set. The image processing unit is thus configured to recognize one or more objects from the image data set, i.e., to classify them as objects. For this purpose, object properties can be predefined that the image data describing the object must fulfill in order for an image to be classified as an object. The results of the processing operation delivered to the data bus therefore advantageously contain an object extracted from the image data set.

[0017] Furthermore, it is advantageous if one of the image processing units is equipped to determine the object's coordinates. These coordinates are preferably referenced to a direction fixed to the missile, for example, the missile's axis. The results available on the data bus therefore advantageously contain the object's coordinates.

[0018] To distinguish a target from other image objects and classify it as such, it is advantageous if one of the image processing units is prepared to perform a classification of the object, where at least one of the available classes is a target. The results available on the data bus therefore advantageously include a classification of the object.

[0019] When calculating steering commands for the autopilot, it is advantageous to know the time interval between the time of image acquisition and the time the main result is provided to the autopilot. This allows the system to know the coordinates of the target at any given time. Furthermore, the calculation of steering commands is significantly simplified if the time interval between these two points in time is always constant. This enables the autopilot to calculate the correct steering commands consistently. Therefore, it is beneficial if data fusion of multiple image datasets forwards the main result to the autopilot after a consistent time interval following image acquisition. Ideally, the length of this time interval should be known.

[0020] It also simplifies the calculation of steering commands if the time intervals between two main results provided to the autopilot are always the same. This can be achieved if the data fusion of multiple image datasets regularly transmits the main results to the autopilot.

[0021] The processing of the image data set into a work result, and the subsequent processing of that work result into a further work result based on it, and so on, can be carried out strictly sequentially. However, greater flexibility in calculating the work results can be achieved if multiple image processing units, for example, all image processing units, have access to the image data set and / or at least the work results of at least one other image processing unit. Maximum data access can be achieved if all image processing units have access to both the image data set and the work results provided by other image processing units.

[0022] To consistently calculate the time between image capture and the delivery of the main result to the autopilot, it is advantageous for each image data set to contain a unique identifier. This can be a timestamp. If the camera inputs the image data sets to the data bus at a known, timed interval, the identifier can also be a number or a code not directly related to a time. In this case, it is beneficial if the timing is also available for data fusion, allowing it to determine the time at which the main result should be delivered to the autopilot based on the identifier.

[0023] To ensure that the time interval between the time of image capture and the transfer of the main result to the autopilot is consistently maintained, the invention provides that a separate image data set is made available for each set of images, with each image data set having a predetermined and equal processing time available on the data bus. At the end of this time period, the data fusion process, according to the invention, closes a processing window for the image data set and / or the resulting work products on the data bus. The main result is then generated from the work products available up to that point and transferred to the autopilot.

[0024] However, it can be useful to allow at least one of the image processing units to continue working on an image dataset and / or a work result. For example, an object classification will not differ from image to image, so calculations on the classification can still be performed even if the processing window for the relevant image dataset has already been closed. The classification can then be added later without disrupting the workflow.

[0025] According to the invention, closing the editing window terminates the processing of the data for an image data set only for a portion of the image processing units, whereas at least one of the image processing units continues processing the data.

[0026] Often, an image dataset is available from which insufficient data can be extracted to guide the missile. For example, no target can be classified. Particularly in such situations, it is advantageous if the data fusion, after closing an editing window, checks whether the results of the image dataset are sufficient to derive predefined, adequately controlled steering commands. If the check fails, the data fusion can discard the results. The missile can then, for example, continue flying unchanged or be guided according to other criteria.

[0027] Depending on the flight situation, determining one of the work results may require particularly high computational effort, while determining another work result requires below-average computational effort. In order to adjust the hardware resources to the necessary computational effort in such a case, it is advantageous if the data fusion system checks the quality of the work results and allocates hardware resources to the image processing units based on this quality.

[0028] With the previously common, rigid timing of the evaluation chain between camera and autopilot, the camera was forced to output its image data sets at fixed intervals. In poor lighting conditions, this could result in an integration time that was too short to obtain sufficiently usable images for steering purposes. The more flexible bus system circumvents this disadvantage. An exposure meter can be advantageously used to check the lighting conditions, for example, based on an image data set output by the camera. To obtain usable image data sets, it is therefore beneficial if the camera's integration time for the recordings is controlled using an exposure meter that evaluates the image data.

[0029] The invention further relates to a missile control unit for a guided missile according to claim 10, comprising a camera. To achieve improved operational capability of the missile control unit, it comprises a data bus connected to the camera, several image processing units on the data bus, data fusion, and an autopilot. The camera is configured to record an environment, generate an image data set from the recording, and make the image data set available on the data bus. The image processing units are configured to process data available on the data bus and make the results available on the data bus. The data fusion is configured to combine the results into a main result, and the autopilot is configured to calculate steering commands for the control surfaces of the guided missile from the main result.

[0030] Depending on the application, the requirements for guided missiles can differ considerably. This can be reflected in the requirements for evaluating the image data sets and thus in the requirements for the image processing units. The invention allows for a flexible response to this, as image processing units are easily interchangeable without requiring modifications to the other components.

[0031] It is therefore advantageous if, in a missile control system with at least two missile control units, these two missile control units are identical with regard to camera, data bus, data fusion and autopilot, and differ from each other in at least one image processing unit.

[0032] The preceding description of advantageous embodiments of the invention contains numerous features, some of which are summarized in several dependent claims. However, the features can also be expediently considered individually and combined into meaningful further combinations, particularly in the case of cross-references between claims, so that a single feature of a dependent claim can be combined with one, several, or all features of another dependent claim. Furthermore, these features can each be combined individually and in any suitable combination with both the method and the apparatus according to the independent claims. Thus, method features can also be considered as properties of the corresponding apparatus unit, and functional apparatus features can also be considered as corresponding method features.

[0033] The properties, features, and advantages of this invention described above, as well as the manner in which they are achieved, will become clearer and more readily understandable in connection with the following description of an exemplary embodiment, which is explained in more detail in conjunction with the drawings. This exemplary embodiment serves to illustrate the invention and does not limit the invention to the combination of features specified therein, including functional features.

[0034] They show: FIG 1 a guided missile for tracking a moving target, FIG 2 a scheme of a data bus with several image processing units connected to it and data fusion before an autopilot, and FIG 3 a scheme with several processing cycles to several image data sets and resulting work results.

[0035] FIG 1 Figure 1 shows a guided missile 2 with a seeker head 4, which contains a camera 6 with optics 8 for imaging an object scene onto a matrix detector 10. To protect the camera 6, the seeker head 4 is closed at the front by a dome with an optical window 12. A missile control unit 14, which is connected to the camera 6 via signal technology, controls the flight of the guided missile 2 by deriving steering commands from image signals from the camera 6 and thereby controlling control fins 16.

[0036] To approach the target, the guided missile 2 was directed to the target, for example by a platform from which it was launched, or it acquired the target independently. Using its camera, the guided missile 2 tracks the target by taking six photographs of the target with its camera pointed at the target, recognizing the target in these photographs, and then flying towards it.

[0037] FIG 2 Figure 14 schematically shows the missile control unit 14 used for control. The camera 6 is connected to a data bus 18, which also connects three image processing units 20, 22, 24, a feedback sensor R, and a data fusion unit DF. The data fusion unit DF is connected to an autopilot AP, which controls actuators (not shown) that move the control surfaces 16.

[0038] During the approach to the target, camera 6 takes a large number of photographs and generates an image data set B from the data of the matrix detector 10 for each photograph (see FIG 3 ), which it makes available on data bus 18. The images can be captured at regular intervals, and the image data sets B can be made available at regular intervals on data bus 18. Each image data set B carries a unique identifier, such as a timestamp or a sequential number. In the example from FIG 3 A sequential number is used, and four image data sets B with the numbers N, N+1, N+2 and N+3, placed directly one after the other on the data bus 18, are shown.

[0039] The image processing units 20, 22, and 24 independently access the data present or provided on data bus 18 and process it into work results, which they then make available again on data bus 18. The work results are in FIG 3 The results are marked with "+" or "-". Each result is identified by the identifier from the image dataset from which it was derived. This allows all results related to a single image dataset to be identified as belonging together. From the results derived from image dataset B, the data fusion process DF generates a main result HE, which the autopilot uses to calculate steering commands for the control surfaces 16 and their actuators.

[0040] At time t(N), camera 6 delivers image data set B(N) to data bus 18. This data set is either fed to all image processing units 20, 22, 24, or they access it when their software requires them to process its data. Image data set B is processed by one or more of the image processing units 20, 22, 24. The processing method can depend on the type of guided missile 2 and / or its mission. In the illustrated embodiment, image data set B can only be processed by image processing unit 20, which is a so-called low-level function and extracts one or more objects from image data set B using image processing methods. Information about the object(s), such as characterizing image data and / or its position in the image, is provided as output by image processing unit 20 on data bus 18.

[0041] These results are acquired by the second image processing unit 22 and processed into further results, such as the object's position relative to the longitudinal axis of the guided missile 2. This information is also made available on data bus 18. The second image processing unit 24 can access the image data and / or the object data and generate a category for the object, for example, whether it is the target being tracked or not. This information is also made available on data bus 18.

[0042] The timing of each processing step can be selected according to the task. Image processing units 20, 22, and 24 each deliver their results to data bus 18, and the other image processing units 20, 22, and 24 retrieve the required data from data bus 18 according to a predefined processing sequence or method. The DF data fusion software module then uses these results to prepare the main result, HE, which is made available to the autopilot.

[0043] To generate control commands for the autopilot in the main result, it is necessary to know the time interval Δt from time t(N) until the time the processed data is made available to the autopilot. Furthermore, it is advantageous if this time interval Δt between the two time points remains constant even for multiple image data sets B. This allows the autopilot (AP) to translate the data into steering commands in a regular and relatively simple manner.

[0044] To ensure that the fixed time interval Δt is adhered to, the data fusion (DF) process closes a time window for processing all data belonging to an image dataset before the time interval Δt expires, thus leaving the DF sufficient time to generate the main result. The processing window δt is therefore shorter than the time interval Δt.

[0045] For each image data set B, the data fusion (DF) waits for the end of the processing window δt and then closes it. The processing window δt is now closed, and the available data can be processed for the autopilot (AP). Closing the processing window δt means that the processing of the data for the respective image data set B is complete. However, it can be selected for which image processing units 20, 22, 24 this applies. In the example from FIG 3 The processing window δt is closed for image processing units 20 and 22, but not for image processing unit 24, because determining the object category can take longer than the processing window δt allows time. Since the image category does not change from image to image, the image category from an older image can still be used later.

[0046] It frequently happens that a recording is of such low quality that not all work results are available at the end of a processing window δt, or even that derivation of steering commands is not possible. This is in FIG 3 depicted.

[0047] FIG 3 Figure 1 shows a scheme with multiple processing cycles for multiple image data sets B and the resulting work outputs. All image processing units 20, 22, and 24 can successfully calculate their work outputs for image data set N. This is shown in Figure 2. FIG 3 This is shown by marking all work results with a "+". For the immediately following image data set N+1, the two image processing units 20 and 22 can successfully produce their work results, but for image processing unit 24, there is insufficient time or data. The work result is either not available or insufficient, which is shown in FIG 3 is marked with a "-".

[0048] In the third processing window δt, which belongs to image data set N+2, even the Low Level Function cannot determine its result, for example, because the object previously identified as the target is not sufficiently well represented in the corresponding image. Accordingly, the Medium Level Function cannot determine any coordinates either. The High Level Function does not process the data for this image data set at all, for example, because it has been instructed by the data fusion function DF to continue calculating the categorization for an earlier image data set B, or because no usable Low Level and / or Medium Level results are available. This is in FIG 3 marked with the crossed-out HLF. All work results are again available in the subsequent processing window δt for image data set N+3.

[0049] If sufficient work results are available, the data fusion (DF) can calculate a sufficient main result (HE) from them, as shown in... FIG 3This is evident. A lack of categorization can still lead to a primary result (+) from which steering commands can be derived because the object is recognized and its coordinates are known. Since it was previously classified as a target, this classification is maintained through a period of missing classification.

[0050] If no main result HE can be determined, the following procedure can be used: Since the timing for the autopilot AP should be maintained, the same information is passed to it for the unsuccessful cycle as for the previous cycle, so that the guided missile 2 continues its current flight maneuver unchanged, for example. The course can then be corrected again with the next successful results. At the corresponding times t(N+i) + Δt, i = 0, 1, 2, ..., the main result is passed to the autopilot AP, and if no current main result is available, the last valid one is passed. For each image data set B, the data fusion DF independently decides whether the results are suitable for controlling the autopilot AP and converts them accordingly if they are suitable for the autopilot.At the closing time of the processing window δt, the existing work results are frozen by the data fusion DF and examined for their suitability for missile control. Depending on the examination results, the data are prepared for the autopilot AP.

[0051] When a window is closed, the data fusion (DF) can send the corresponding closing information to data bus 18, so that the image processing units 20, 22, 24 can end their work on the corresponding image data set B. The individual image processing units 20, 22, 24 can independently decide which of the several image data sets B simultaneously present on data bus 18 within a processing window δt they will work on.

[0052] The data fusion unit (DF) performs a coordinating function. It not only opens and closes the processing windows δt but can also monitor the activity of the other image processing units 20, 22, and 24. For example, it can shut down or otherwise control unsuccessful image processing units 20, 22, and 24 to free up processing power. This is because the hardware of the individual image processing units 20, 22, and 24 does not need to be strictly separated; they can run on shared hardware that provides the necessary resources. Hardware can also be allocated according to the resource requirements of the individual image processing units 20, 22, and 24. For example, processes that are essential for controlling the autopilot (AP) are allocated more hardware to ensure they deliver results even in challenging image situations.

[0053] This new system has the advantage that the individual image processing units 20, 22, 24 can be easily replaced without significantly affecting the operation of the other image processing units 20, 22, 24. The system's modularity is significantly increased, and modifications can be easily implemented.

[0054] The modularity can be used for the following applications. For example, simple image processing units are available that group together 2 × 2, 4 × 4, or 8 × 8, etc., pixels. The image transmitted from camera 6 to data bus 18 thus becomes increasingly blurry. Another image processing unit can include a Laplace filter, i.e., a point detector. If the Laplace filter is applied to a single pixel, very small pixels can be recognized as such. The missile can then detect very distant targets. However, if the Laplace filter is applied exclusively to grouped pixels, for example, only to grouped 2 × 2 pixels, it will not recognize individual significant pixels, but will be more reliable in tracking closer targets.In this way, one guided missile 2 can be equipped with detection capabilities for distant or smaller targets, and another guided missile 2 with good detection capabilities for closer or larger targets, for example, according to mission specifications for guided missile 2. This modularity is very easy to implement, as only the corresponding processing unit needs to be inserted.

[0055] The new system has the further advantage that the usual clocking by camera 6 can also be modified. For example, a feedback sensor R is inserted, which provides asynchronous feedback. The feedback sensor R can be an exposure meter that checks the amount of radiation from the data of the image data set B. If an image is too dark or too bright, this brightness control sends corresponding data to the data bus 18 to camera 6, which can then adjust the integration time of its sensor accordingly. If the integration time exceeds the previous clocking, the clocking of camera 6 can be increased. This is unproblematic as long as the size of the time window δt is not changed. Reference symbol list

[0056] 2 Guided missile 4 Seeker head 6 Camera 8 Optics 10 Matrix detector 12 Window 14 Missile control unit 16 Control fin 18 Data bus 20 Image processing unit 22 Image processing unit 24 Image processing unit B Image data set AP Autopilot DF Data fusion HE Main result R Sensor

Claims

1. Image processing method for determining steering commands for control fins (16) of a guided missile (2), in which - a camera (6) records an environment, generates an image data set (B) from the recording and makes the image data set (B) available on a data bus (18), - a plurality of image processing units (20, 22, 24) connected to the data bus (18) process data available on the data bus (18) and make their working results available on the data bus (18), wherein the data are the image data set (B) or a working result of one of the image processing units (20, 22, 24) that results from the image data set (B) or a working result that is captured by one of the image processing units (20, 22, 24), results from the image data set and is processed further, - a data fusion (DF) combines the working results to form a main result (HE), - an autopilot (AP) calculates steering commands for the control fins (16) from the main result (HE), - one image data set (B) is made available for a plurality of recordings, characterized in that - each image data set (B) is available for a predetermined and equal time period for processing on the data bus (18), - the data fusion (DF) closes a temporal processing window for the image data set (B) and the resulting working results on the data bus (18) for the image processing units (20, 22), wherein the closing of the processing window ends the processing of the data for the image data set (B) and / or for a resulting working result only for some of the image processing units (20, 22), whereas at least one of the image processing units (24) continues the processing of the data for the image data set (B) and / or for a resulting working result, wherein the main result (HE) is created from the previously available working results and is transferred to the autopilot (AP).

2. Method according to Claim 1, characterized in that the working results contain an object extracted from the image data set (B), the coordinates of the object and / or a classification of the object.

3. Method according to Claim 1 or 2, characterized in that the data fusion (DF) for a plurality of image data sets (B) forwards the main result (HE) to the autopilot (AP) after always the same time period (Δt) after the recording.

4. Method according to one of the preceding claims, characterized in that the data fusion (DF) for a plurality of image data sets (B) regularly forwards the main result (HE) to the autopilot (AP) in a clocked fashion.

5. Method according to one of the preceding claims, characterized in that all image processing units (20, 22, 24) have access to both the image data set (B) and the working results provided by other image processing units (20, 22, 24).

6. Method according to one of the preceding claims, characterized in that each image data set (B) contains a unique identification tag.

7. Method according to one of the preceding claims, characterized in that, after closing a processing window, the data fusion (DF) checks whether the working results for an image data set (B) are sufficient to derive predefined sufficient steering commands, and, if the check result is negative, discards the working results.

8. Method according to one of the preceding claims, characterized in that the data fusion (DF) checks the working results for quality and allocates hardware resources to the image processing units (20, 22, 24) running on common hardware based on the quality, wherein the hardware is allocated in the manner in which the individual image processing units (20, 22, 24) require resources.

9. Method according to one of the preceding claims, characterized in that the integration time of the camera (6) for the recordings is controlled using an exposure tester that evaluates the image data.

10. Missile control unit (14) for a guided missile (2), having - a camera (6), - a data bus (18) connected to the camera (6), - a plurality of image processing units (20, 22, 24) on the data bus (18), - a data fusion (DF) and an autopilot (AP), wherein - the camera (6) is prepared to record an environment, to generate an image data set (B) from the recording and to make the image data set (B) available on the data bus (18), - the image processing units (20, 22, 24) are prepared to process data available on the data bus (B) and to make the working results available on the data bus (18), wherein the data are the image data set or a working result of one of the image processing units (20, 22, 24) that results from the image data set (B) or a working result that is captured by one of the image processing units (20, 22, 24), results from the image data set and is processed further, - the data fusion (DF) is prepared to combine the working results to form a main result (HE), and - the autopilot (AP) is prepared to calculate steering commands for control fins (16) of the guided missile (2) from the main result (HE), characterized in that - the data fusion (DF) is prepared, after the end of a predetermined and equal time period which is available to a respective image data set of a plurality of recordings for processing on the data bus, to close a temporal processing window for the image data set (B) and the resulting working results on the data bus (18) for the image processing units (20, 22), wherein the closing of the processing window ends the processing of the data for the image data set (B) and / or for a resulting working result only for some of the image processing units (20, 22), whereas at least one of the image processing units (24) continues the processing of the data for the image data set (B) and / or for a resulting working result, and - the data fusion (DF) is prepared to create the main result (HE) from the previously available working results and to transfer it to the autopilot (AP) after such closing of the processing window.

11. Missile control system having at least two missile control units (14) according to Claim 10, characterized in that the two missile control units (14) are identical with regard to the camera (6), data bus (18), data fusion (DF) and autopilot (AP) and differ from each other in at least one image processing unit (20, 22, 24) with regard to the requirements for the evaluation of the image data sets (B).