Different frame rate settings

Abstract

The present invention relates to different frame rate settings. A technique for setting the frame rate of a camera is provided. The camera includes a plurality of image sensors arranged to capture images to be stitched into a panoramic image of a scene. Information is obtained about a corresponding angle between an optical axis of each of the plurality of image sensors and a direction along which a straight path structure to be monitored by the camera extends through the scene. Based on the angle, the plurality of image sensors are divided into at least two groups. All angles of the image sensors within each of the at least two groups are part of its own continuous angle interval. A frame rate is set for each of the at least two groups. The frame rate decreases from the group with the highest continuous angle interval to the group with the lowest continuous angle interval.

Description

Technical Field

[0001] Embodiments presented herein relate to methods, controllers, computer programs, and computer program products for setting a frame rate of a camera. Background Art

[0002] Generally speaking, a panoramic camera provides wide-area coverage of a scene (hereinafter referred to as a panoramic image) using only one camera. A panoramic camera can be used to monitor scenes consisting of physical structures, activities, and detect events over a large area, track the flow of people, vehicles, or other types of objects, and improve area management. Using such a panoramic camera instead of multiple cameras with narrow area coverage can simplify installation and reduce costs.

[0003] A panoramic camera includes two or more or even multiple image sensors. Each image sensor is arranged to have its own field of view, but the fields of view of two adjacently placed image sensors in a panoramic camera overlap slightly. This enables images from different image sensors to be stitched together to form a single panoramic image. In this regard, image stitching or photo stitching is the process of combining multiple images with overlapping fields of view to produce a segmented panoramic or high-resolution image. Image stitching typically involves image registration, calibration, and blending.

[0004] Panoramic images captured over time may be combined into video clips. Such video clips may then be subjected to further analysis (such as object detection and recognition, and / or provided to a user for display). The greater the number of image sensors used to capture the images to be stitched, and the higher the frame rate used by the image sensors, the greater the amount of data generated, and the higher the computational requirements and power consumption for creating and processing video clips from the panoramic images. High power consumption may increase heating of the camera.

[0005] However, there may be situations where available data storage is limited, where computational requirements (e.g., for decoding) need to be kept low, where power consumption needs to be limited, and / or where bit rate or bandwidth is required for communication between the different entities involved in creating video segments from the panoramic images. One way to reduce these requirements is to reduce the resolution of the images, or to use a compression scheme with a relatively high compression ratio. However, this may result in a loss of detail in the scene to be monitored, thereby compromising safety. Summary of the invention

[0006] The purpose of the embodiments of this article is to solve the above problems.

[0007] Generally speaking, according to the inventive concepts disclosed herein, the frame rate is set differently for different image sensors of a camera.

[0008] According to a first aspect, the inventive concept is defined by a method for setting a camera frame rate. The camera includes a plurality of image sensors arranged to capture images to be stitched into a panoramic image of a scene. Each of the plurality of image sensors has a corresponding optical axis. The method is performed by a controller. The method includes obtaining information about a corresponding angle between an optical axis of each of the plurality of image sensors and a direction along which a straight path structure to be monitored by the camera extends through the scene. The method includes dividing the plurality of image sensors into at least two groups according to angles. All angles of the image sensors within each of the at least two groups are part of its own continuous angle interval. The method includes setting a frame rate for each of the at least two groups. The frame rate decreases from the group with the highest continuous angle interval to the group with the lowest continuous angle interval.

[0009] According to a second aspect, the inventive concept is defined by a controller for setting a camera frame rate. The camera includes a plurality of image sensors arranged to capture images to be stitched into a panoramic image of a scene. Each of the plurality of image sensors has a corresponding optical axis. The controller includes a processing circuit. The processing circuit is configured to cause the controller to obtain information about a corresponding angle between an optical axis of each of the plurality of image sensors and a direction along which a straight path structure to be monitored by the camera extends through the scene. The processing circuit is configured to cause the controller to divide the plurality of image sensors into at least two groups according to angles. All angles of the image sensors within each of the at least two groups are part of its own continuous angle interval. The processing circuit is configured to cause the controller to set a frame rate for each of the at least two groups. The frame rate decreases from the group with the highest continuous angle interval to the group with the lowest continuous angle interval.

[0010] According to a third aspect, the inventive concept is defined by a system. The system comprises a controller according to the second aspect and a camera arranged in the second aspect.

[0011] According to a fourth aspect, the inventive concept is defined by a computer program for setting a frame rate of a camera, the computer program comprising computer program code which, when run on a controller, causes the controller to perform the method according to the first aspect.

[0012] According to a fifth aspect, the inventive concept is defined by a computer program product, the computer program product comprising the computer program according to the fourth aspect and a computer readable storage medium storing the computer program. The computer readable storage medium may be a non-transitory computer readable storage medium.

[0013] Advantageously, these aspects address the above-mentioned issues by requiring less data storage and / or requiring less computational requirements for generating panoramic images.

[0014] Advantageously, by reducing the frame rate of some sensors relative to other sensors, computational resources and the (average) bitrate of the final video stream can be saved.

[0015] Advantageously, the computational resources saved and frame rate reduced thereby may in turn reduce the overall power and energy consumption of the camera, or allow the image resolution of the camera to be increased.

[0016] Other objectives, features and advantages of the included embodiments will become apparent from the following detailed disclosure, the attached dependent claims as well as the drawings.

[0017] Generally, unless otherwise explicitly defined herein, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field. Unless otherwise explicitly stated, all references to "one / the element, device, component, means, module, step, etc." should be interpreted openly as referring to at least one instance of an element, device, component, means, module, step, etc. Unless explicitly stated otherwise, the steps of any method disclosed herein do not have to be performed in the exact order disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention is now described by way of example with reference to the accompanying drawings, in which:

[0019] Figure 1 is a schematic diagram illustrating a system according to an embodiment;

[0020] Figure 2 schematically illustrates a camera according to an embodiment;

[0021] Figure 3 Schematically illustrates a first scenario according to an embodiment;

[0022] Figure 4 schematically illustrates a second scenario according to an embodiment;

[0023] Figure 5 shows a panoramic image according to an embodiment;

[0024] Figure 6 is a flow chart of a method according to an embodiment;

[0025] Figure 7 schematically illustrates readout from an image sensor according to an embodiment;

[0026] Figure 8 schematically illustrates a grouping of image sensors according to an embodiment;

[0027] Fig. 9 is a schematic diagram showing functional units of a controller according to an embodiment; and

[0028] Fig.10 An example of a computer program product including a computer-readable storage medium according to an embodiment is shown. DETAILED DESCRIPTION

[0029] The inventive concept will now be described more fully below with reference to the accompanying drawings, in which specific embodiments of the inventive concept are shown. However, the inventive concept can be implemented in a variety of different forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that the disclosure will be thorough and complete and will fully convey the scope of the inventive concept to those skilled in the art. Throughout the description, the same reference numerals refer to the same elements. Any steps or features illustrated by dashed lines should be considered optional.

[0030] Figure 11 is a schematic diagram illustrating a scene 100 in which the embodiments presented herein can be applied. System 110 includes a camera 114 and a controller 112 for controlling camera 114. Camera 114 further includes image sensors 116a, 116b, 116c. As will be further disclosed below, camera 114 may include more than image sensors 116a, 116b, 116c. Generally speaking, camera 114 is configured to capture image frames depicting scenes within a field of view with the aid of image sensors 116a, 116b, 116c. In some examples, camera 114 is a digital camera. In some examples, camera 114 is capable of mechanical or digital panning, tilting and / or zooming (PTZ). System 110 is configured to communicate with a user interface 130 for displaying captured image frames. Furthermore, the system 110 is configured to encode the image so that it can be decoded using any known video coding standard, such as any of the following standards: High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2; Advanced Video Coding (AVC), also known as H.264 and MPEG-4 Part 10; Versatile Video Coding (VVC), also known as H.266, MPEG-I Part 3, and Future Video Coding (FVC); VP9, ​​VP10 AOMedia Video1 (AV1), Motion JPEG (M-JPEG or MJPEG, where JPEG is an abbreviation for Joint Camera Experts Group), to name a few examples. In this regard, the encoding can be performed directly in conjunction with the image sensor 116a, 116b, 116c that captures the image frame, or performed at another entity and then stored at least temporarily in the database 120. The system 110 may include more entities, functions, nodes, and devices (as represented by the device 118). Examples of other such entities, functions, nodes, and devices are computing devices, communication devices, servers, sensors, positioning devices, etc. At reference numeral 140, an example connection between the system 110 and the user interface 130 is illustrated. The connection 140 can be wired, wireless, or partially wired and partially wireless. A user 180 can interact with the user interface 130. It should be understood that the user interface 130 is at least partially a visual user interface in that it is configured to display image frames captured by the image sensors 116a, 116b, 116c of the camera 114.

[0031] Next, we will refer to Figure 2 Other aspects of the camera 114 are disclosed. Figure 2 In the illustrative example of , the camera 114 includes K image sensors 116a, 116b, ..., 116k, ..., 116K. Each of the image sensors 116a:116K is configured to capture a a ,δb , …, δ k , …, δ K Each image sensor also has a vertical field of view (not shown). Image sensors 116a:116K may collectively span a horizontal field of view between 120° and 185°. To avoid confusion, Figure 2 In the illustrative example of , the image sensors 116a: 116K collectively span a horizontal field of view of approximately 220°. The angular bisector of each field of view defines an optical axis 210a, 210b, ..., 210k, ..., 210K of each image sensor 116a: 116K. The camera 114 has a camera axis 220. In some examples, the camera axis 220 defines the aperture aiming direction of the camera 114. In some examples, the camera axis 220 is an axis about which the camera 114 can pivot when mounted. In some examples, the camera axis 220 is given by the average of the optical axes 210a: 210K of all image sensors 116a: 116K. In general, it is assumed that the camera 114 is arranged to monitor a straight path structure from an elevated position. Non-limiting examples of such a straight path structure are: a vehicle road, a sidewalk, an office corridor, a railway track, an airport taxiway, an airport runway, a ship waterway, a portion of a sports field, or a runway. Furthermore, the image sensors 116a: 116K are arranged to capture images to be stitched into a panoramic image of the scene. Thus, it is assumed that the scene comprises one or more rectilinear path structures to be monitored.

[0032] Figure 3 A first scenario 300 of the inventive concepts disclosed herein is schematically illustrated. According to the first scenario 300, a camera 114 is used to monitor a straight path structure 310 in the form of a vehicle road. The camera 114 may be mounted or fixed to a pole, pillar, building, or other vertically extending physical structure. The camera 114 may be 3-5 meters above the surface of the straight path structure 310. In this regard, the camera 114 may be tilted downward so that the straight path structure 310 is captured within the vertical and horizontal fields of view of the image sensor. Two objects 320a, 320b in the form of cars are depicted traveling along the vehicle road. The straight path structure 310 extends along a direction D. In Figure 3 In the illustrative example of , the camera axis 220 is therefore perpendicular to the direction D, and the straight path structure 310 extends through the scene 300 along the direction D. The camera 114 includes six image sensors. The optical axes of the image sensors are shown as radiating from the camera 114. The optical axis of each of the image sensors has its own angle relative to the direction D. These angles are Figure 3α1, α2, α3, α4, α5, α6 are illustrated in FIG. Therefore, the angles α1, α2, α3, α4, α5, α6 represent the angles between the optical axes of the six image sensors and the direction D. That is, α1 is the angle between the optical axis of the leftmost image sensor and the direction D, α2 is the angle between the optical axis of the second image sensor from the leftmost and the direction D, and α6 is the angle between the optical axis of the rightmost image sensor and the direction D, and so on. For the inventive concepts disclosed herein, the signs of the angles will be insignificant, so it is assumed hereinafter that the angles are represented only by their absolute values. That is, in scene 300, α1=α6 (within some margin), and so on.

[0033] Figure 4 A second scenario 400 of the inventive concepts disclosed herein is schematically illustrated. According to the second scenario 400, the camera 114 is used to monitor two straight path structures 410a, 410b, both of which are in the form of vehicle roads. The camera 114 can be mounted or fixed on a column, a pillar, a building or other vertically extending physical structure. Figure 4 In the illustrative example of FIG. 4 , the camera 114 is mounted on the bridge 430 and may be approximately 5 meters above the surface of the straight path structures 410a, 410b. In this regard, the camera 114 may be tilted downward so that the straight path structures 410a, 410b are captured within the vertical and horizontal fields of view of the image sensor. Four objects 420a, 420b, 420c, 420d in the form of cars are depicted traveling along the vehicle road. Both the straight path structures 410a, 410b extend in a direction D. In FIG. Figure 4 In the illustrative example of , the camera axis 220 is therefore perpendicular to the direction D, and the straight path structures 410a, 410b extend through the scene 400 along the direction D. As in the scene 300, the camera 114 includes six image sensors. The optical axes of the image sensors are shown as irradiated from the camera 114. The optical axis of each of the image sensors has its own angle relative to the direction D. As in the scene 300, these angles are Figure 4 α1, α2, α3, α4, α5, α6 are illustrated in FIG. Therefore, angles α1, α2, α3, α4, α5, α6 represent angles between the optical axes of the six image sensors and the direction D. For example, in scene 300, α1 is the angle between the optical axis of the leftmost image sensor and the direction D, α2 is the angle between the optical axis of the second image sensor from the leftmost and the direction D, and α6 is the angle between the optical axis of the rightmost image sensor and the direction D, and so on.

[0034] Therefore, due to the different geometries with respect to the orientation of the camera axis 220 relative to the direction D in the two different scenes 300, 400, the angle α is 200° compared to the scene 300. k；k＝1…KEach of takes a different value in scene 400. For scene 300, the optical axis of the image sensor at the edge has the smallest angle α1, α6, and the image sensor in the center has the largest angle α3, α4. That is, for scene 300: (α1, α6)<(α2, α5)<(α3, α4). For scene 400, the situation is opposite, so the optical axis of the image sensor at the edge has the largest angle α1, α6, and the image sensor in the center has the smallest angle α3, α4. That is, for scene 400: (α3, α4)<(α2, α5)<(α1, α6).

[0035] Although Figure 3 A scene 300 is illustrated, wherein the camera axis 220 is perpendicular to a direction D, a linear path structure 310 extends through the scene 300 along the direction D, and Figure 4 A scene 400 is illustrated in which the camera axis 220 is parallel to the direction D and the straight path structures 410a, 410b extend through the scene 300, 400 along the direction D, but the embodiments disclosed herein are not limited to these two types of scenes 300, 400. The embodiments disclosed herein also include other relationships between the camera axis 220 and the straight path structures, such as the camera axis 220 is neither perpendicular nor parallel to the direction D and the straight path structures 410a, 410b extend through the scene 300, 400 along the direction D, but the relationship takes another value.

[0036] Figure 5 An illustration of a real-world panoramic image 500 according to an implementation of the camera 114 is provided in an artistic form. The panoramic image 500 is thus stitched together from images captured by the image sensors 116a: 116K of the camera 114. Two straight path structures 510a, 510b are shown in direction D with respect to the vehicle road. The camera 114 is thus arranged to monitor the vehicle road. Four cars representing objects traveling along the vehicle road are identified with reference numerals 520a, 520b, 520c, 520d.

[0037] Refer again Figure 2 , each image sensor 116a: 116K has a unique optical axis 210a: 210K. This ensures that the image sensors 116a: 116K face different directions so as to capture partially overlapping portions of the scene. This partial overlap enables images from adjacently placed image sensors to be stitched together. Repeating this stitching for all image sensors 116a: 116k enables the creation of Figure 5. In more detail, for this purpose, it is assumed that the image sensor uses a cylindrical projection. This means that a certain number of pixels along the horizontal image axis corresponds to a certain number of degrees, regardless of whether these certain number of pixels are in the middle of the image or at the edge of the image. In this regard, as can be seen in the panoramic image 500, the straight path structures 510a, 510b appear not to be completely straight, but rather appear to have a curved or bent appearance. This is due to the same optical effect of the camera 114, which typically causes the horizon to appear to have a curved shape in a panoramic image.

[0038] Now consider again scene 300. Since the monitored vehicle road defines a straight path structure 310, the distance traveled by an object traveling along the vehicle road at a constant speed will vary when the distance is measured in pixels per unit time. First, the distance varies by a factor that depends on the angle relative to the camera axis 220 at which the object is located along the vehicle road. In scene 300, the angle for object 320b is greater than the angle for object 320a. The factor varies with the cosine of the angle. Second, the distance varies because the left and right edges of the vehicle road generated angles α1, α6 monitored by the image sensor will be physically farther from the camera 114 than the midpoint of the vehicle road generated angles α3, α4 monitored by the image sensor. The variation depends on the distance between each portion of the vehicle road and the camera 114. Corresponding reasons also apply to scene 400. The embodiments disclosed herein are based on exploiting these facts when setting the frame rate of the image sensors 116a: 116K.

[0039] In some embodiments, the frame rates of the image sensors 116a: 116K are set so that an image sensor that captures an object that moves relatively few pixels from one image to the next over time at a given constant speed is given a lower frame rate (for a given frame rate) than an image sensor that captures the same object that moves relatively more pixels from one image to the next at the same given constant speed. That is, since the number of pixels that an object moves from one image to the next depends on the frame rate (i.e., the number of frames per second), for image sensors with the same frame rate, the movement of the objects in terms of pixels is compared here to produce a fair comparison.

[0040] Consider scene 300, and assume that each object 320a, 320b moves at a constant speed. Vehicle or object 320a is near the midpoint of the vehicle road, and vehicle or object 320b is near the left edge of the vehicle road monitored by camera 114. Therefore, object 320a moves more pixels from one image to the next over time at the same given constant speed compared to object 320b. The inventors of the inventive concepts disclosed herein have recognized that this means that the frame rate of the image sensor that captures the image of object 320b can be set to be lower than the frame rate of the image sensor that captures the image of object 320a without losing information about object 320b. The user 180 viewing the panoramic image corresponding to scene 300 will still be able to follow the movement of object 320b without experiencing visual artifacts (e.g., any of the vehicles leaping forward between consecutive images).

[0041] For scene 300, this means that the frame rate of the image sensor with angles α1, α6 can therefore be lower than the frame rate of the image sensor with angles α3, α4. For scene 400, the opposite is true; the frame rate of the image sensor with angles α1, α6 can be higher than the frame rate of the image sensor with angles α3, α4. This is because in scene 400, vehicle or object 420a is closer to camera 114 than vehicle or object 420c.

[0042] Therefore, embodiments disclosed herein relate to techniques for setting the frame rate of the camera 114. To achieve the techniques, a controller 112, a method performed by the controller 112, a computer program product including code (e.g., in the form of a computer program that, when executed on the controller, causes the controller to perform the method) is provided.

[0043] Figure 6 is a flow chart illustrating an embodiment of a method for setting a frame rate of a camera 114. The method is performed by a controller 112. The method is advantageously provided as a computer program. As disclosed above, the camera 114 comprises a plurality of image sensors 116a: 116K arranged to capture images to be stitched into a panoramic image 500 of a scene 300, 400. As disclosed above, each of the plurality of image sensors 116a: 116K has a respective optical axis 210a: 210K.

[0044] S106: The controller 112 obtains a corresponding angle α between the optical axis 210a: 210K of each of the plurality of image sensors 116a: 116K and the direction D k；k＝1…K The linear path structure 310, 410a, 410b, 510a, 510b to be monitored by the camera 114 extends through the scene 300, 400 along this direction D. In this regard, it is assumed that 0°≤αk；k＝1…K ≤90°.

[0045] Essentially, the frame rate of the image sensor 116a: 116K is based on the angle α k；k＝1…K Therefore, the image sensors 116a: 116K are set according to the angle α k；k＝1…K The values ​​are divided into groups.

[0046] S108: The controller 112 divides the plurality of image sensors 116a: 116K into at least two groups 820a: 820d. k；k＝1…K , the plurality of image sensors 116a: 116K are divided into groups 820a: 820d. All angles α of the image sensors 116a: 116K in each of at least two groups 820a: 820d are k；k＝1…K is part of its own continuous angular interval 810a:810e. Figure 8 Such examples are disclosed and therefore are included here for ease of reference. Figure 8 The reference numerals used in the drawings.

[0047] Essentially, the frame rate of the image sensor 116a: 116K is based on the angle α k；k＝1…K To set the frame rate so that the angle α k；k＝1…K decreases with the decrease of .

[0048] S110: The controller 112 sets a frame rate for each of the at least two groups 820a:820d. The frame rate decreases from the group 820b having the highest continuous angle interval 810b, 810e to the group 820a, 820c having the lowest continuous angle interval 810a, 810c. That is, the image sensor in the group 820b having the highest continuous angle interval 810b, 810e has the highest frame rate, and the image sensor in the group 820a, 820c having the lowest continuous angle interval 810a, 810c has the lowest frame rate.

[0049] Embodiments related to further details of setting the frame rate of the camera 114 performed by the controller 112 will now be disclosed.

[0050] As outlined above, there may be different triggers of the controller 112 to set the frame rate of the camera 114 .

[0051] In general, assuming that the camera 114 will be fixedly mounted (and therefore not moved) to monitor the straight path structure 310, 410a, 410b, 510a, 510b, and assuming that the straight path structure 310, 410a, 410b, 510a, 510b does not change, then after the camera 114 is deployed, only the frame rate of the camera 114 needs to be set. The frame rate of the camera 114 can even be set as part of the factory settings.

[0052] In a further aspect, a frame rate is set upon detecting the presence of a straight path structure 310, 410a, 410b, 510a, 510b and / or upon detecting movement of the camera 114 relative to the detected straight path structure 310, 410a, 410b, 510a, 510b. Detecting a straight path structure 310, 410a, 410b, 510a, 510b (including direction D) may be accomplished by performing image analysis of an image sequence to detect an actual straight path structure 310, 410a, 410b, 510a, 510b, such as a road structure (and inferring direction D therefrom), or detecting a moving object 320a, 320b, 420a: 420d, 520a: 520d having a common straight direction of movement (which defines direction D). Movement of the camera 114 relative to the linear path structure 310, 410a, 410b, 510a, 510b may be due to weather conditions, wherein the orientation of the camera 114 is affected or impaired by weather conditions such as wind, snow, rain, etc. Therefore, in some embodiments, the controller 112 is configured to perform (optional) step S102.

[0053] S102: The controller 112 detects the presence of the straight path structure 310, 410a, 410b, 510a, 510b and / or the movement of the camera 114 relative to the straight path structure 310, 410a, 410b, 510a, 510b by performing object detection and recognition in the image.

[0054] Then, in response to S102 having been executed, the frame rate is set.

[0055] In other aspects, the frame rate is set based on a request from the user 180. Thus, in some embodiments, the controller 112 is configured to perform (optional) step S104.

[0056] S104 : The controller 112 receives user input to set the frame rate of the camera 114 and / or reduce the computational requirements of the camera 114 .

[0057] Then, in response to S104 having been executed, the frame rate is set.

[0058] There may be different ways to group the image sensors 116a: 116K and thus set the frame rate of the image sensors 116a: 116K. Different embodiments related thereto will now be described in turn.

[0059] In some embodiments, each of the frame rates takes a value from a set of predefined frame rate values. In some non-limiting and illustrative examples, such a set of predefined frame rate values ​​includes values ​​of 15 frames per second, 30 frames per second, 45 frames per second, and 60 frames per second. However, other values ​​are also possible. It should be noted here that not all values ​​in the set of predefined frame rate values ​​need to be used. This depends on how many groups of image sensors 116a: 116K are selected.

[0060] In some aspects, image sensors at the periphery of the camera 114 (thus covering the left and right edges of the horizontally extending panoramic image 500) have a lower frame rate than image sensors in the middle. Specifically, in some embodiments, the frame rate of any given image sensor 116k in the plurality of image sensors 116a: 116k decreases as the difference between the optical axis 210k and the camera axis 220 of any given image sensor 116k increases. This may apply to the scene 300, and vice versa to the scene 400.

[0061] Since different frame rates are set for different image sensors 116a: 116K, this means that some image sensors 116a: 116K will generate more images than other image sensors 116a: 116K. Therefore, there may be different ways to select which images to stitch together to form the panoramic image 500.

[0062] In some aspects, the panoramic image 500 is composed of at least two images captured from different image sensors 116a: 116K at the same point in time. This enables the images read out from the image sensors 116a: 116k to be synchronized in time despite the image sensors 116a: 116k having different frame rates.

[0063] This synchronous reading out of images from the image sensors 116a:116k is also possible when the frame rate of the first group of the at least two groups 820a:820d is a multiple of the frame rate of the second group of the at least two groups 820a:820d. For example, assuming that the frame rate of the image sensor in the first group is f1, and the frame rate of the image sensor in the second group is f2, then f1=c·f2, where c≥2 is an integer.

[0064] Furthermore, in this regard, images captured by image sensors 116a:116K in a first group of at least two groups 820a:820d may be captured at a first group of time opportunities, and images captured by image sensors 116a:116K in a second group of at least two groups 820a:820d may be captured at a subset of the first group of time opportunities. This situation will be illustrated below. For purposes of illustration, assume that there are three image sensors, identified as "image sensor 1," "image sensor 2," and "image sensor 3." In this illustrative example, there are therefore three groups of image sensors, each with one image sensor. It is further assumed that readout is performed from image sensor 1 at times t1 and t3, readout is performed from image sensor 2 at times t1, t2, t3, and t4, and readout is performed from image sensor 3 at times t2 and t4, where t1, t2, t3, and t4 are four equidistant points in time. Figure 7 700 in FIG. 1 schematically illustrates when along the timeline a readout is performed from each of the three image sensors. Thus, four readouts will be performed as follows:

[0065] Read out at t1: Image sensor 1 + Image sensor 2;

[0066] Read out at t2: image sensor 2 + image sensor 3;

[0067] Read out at t3: Image sensor 1 + Image sensor 2; and

[0068] Read out at t4: Image sensor 2 + Image sensor 3.

[0069] Therefore, the frame rate of image sensor 2 can be set to twice the frame rate of image sensor 1 and image sensor 3. This will reduce the number of readouts performed at each time point from three to two. This in turn enables the peak power consumption of camera 114 to be reduced.

[0070] Next, we will refer to Figure 8 Other aspects of how to group image sensors 116a: 116K are disclosed. Figure 8 (a) Figure 8 (b) and Figure 8 (c) schematically illustrates different examples of image sensors (indicated by their angles α1, α2, α3, α4, α5, α6) that can be grouped into different groups 820a, 820b, 820c, 820d according to consecutive angular intervals 810a, 810b, 810c, 810d, 810e. Adjacent consecutive angular intervals are separated by threshold angle values ​​β1, β2, β3.

[0071] from Figure 8(a) Beginning, assume that α1 = α6, α2 = α5, and α3 = α4. There are two consecutive angular intervals 810a, 810b separated by a threshold angle value β1. Thus, image sensors with angles α1, α2, α5, α6 define a first group 820a, and image sensors with angles α3, α4 define a second group 820b.

[0072] exist Figure 8 In (b), it is also assumed that α1=α6, α2=α5 and α3=α4. There are three consecutive angular intervals 810c, 810d, 820e separated by threshold angle values ​​β2 and β3. Therefore, the image sensors with angles α1, α6 define a third group 820c, the image sensors with angles α2, α5 define a fourth group 820d, and the image sensors with angles α3, α4 define a third group 820b.

[0073] exist Figure 8 In (c), all angles α1, α2, α3, α4, α5, and α6 are unique. Figure 8 As shown in (a), there are two consecutive angular intervals 810a, 810b separated by a threshold angle value β1. Therefore, image sensors with angles α1, α2, α5, α6 define a first group 820a, and image sensors with angles α3, α4 define a second group 820b.

[0074] Figure 8 Given by selecting a threshold angle value β between the angle intervals 810a:810e j；j＝1…J Thus, in some embodiments, consecutive angular intervals 810a:810e are separated by a plurality of threshold angle values ​​β j；j＝1…J The number of groups 820a:820d into which the plurality of image sensors 116a:116K are divided depends on the threshold angle value β. j；j＝1…J The number of

[0075] Figure 8 (a) and 8(b) can represent scene 300, while Figure 8 (c) may represent a scene similar to scene 300 , but where camera axis 220 is not completely perpendicular to direction D along which straight path structure 310 extends through scene 300 .

[0076] With further regard to angles α1, α2, α3, α4, α5, α6, when having a calibrated camera 114, it is assumed that these values ​​are known. There may be different ways to calibrate the camera 114. However, how to calibrate the camera 114 is beyond the scope of this disclosure. In some examples, once the camera axis 220 and the direction D are known, the values ​​of α1, α2, α3, α4, α5, α6 can be determined based on image analysis of the scene 300, 400. This is possible because the angle at which the image sensors 116a: 116K are pointed relative to the camera axis 220 (e.g., determined by the angle δ a ,δ b , …, δ k , …, δ K ) and the projections used by the image sensors 116a:116K are assumed to be known. How to determine the direction D has been disclosed above.

[0077] Fig. 9 Components of the controller 112 according to an embodiment are schematically illustrated in terms of a number of functional units. Processing circuitry 910 is provided using any combination of one or more suitable central processing units (CPUs), multiprocessors, microcontrollers, digital signal processors (DSPs), etc., which is capable of executing programs stored in a computer program product 1010 (e.g., Fig.10 ) (e.g., in the form of storage medium 930). Processing circuit 910 may further be provided as at least one application specific integrated circuit (ASIC) or field programmable gate array (FPGA).

[0078] In particular, the processing circuit 910 is configured to cause the controller 112 to perform a set of operations or steps as disclosed above. For example, the storage medium 930 may store the set of operations, and the processing circuit 910 may be configured to retrieve the set of operations from the storage medium 930 so that the controller 112 performs the set of operations. The set of operations may be provided as a set of executable instructions.

[0079] Thus, the processing circuit 910 is thereby arranged to perform the methods disclosed herein. The storage medium 930 may also include permanent storage, for example, which may be any one or combination of magnetic storage, optical storage, solid-state storage, or even remotely mounted storage. The controller 112 may further include a communication interface 920, which is configured at least for communicating with other entities, functions, nodes, and devices (such as Figure 114 and other entities). As such, the communication interface 920 may include one or more transmitters and receivers including analog and digital components. The processing circuit 910 controls the overall operation of the controller 112 by, for example, sending data and control signals to the communication interface 920 and the storage medium 930, by receiving data and reports from the communication interface 920, and by receiving data and instructions from the storage medium 930. Other components of the controller 112 and related functions are omitted so as not to obscure the concepts presented herein.

[0080] Fig.10 An example of a computer program product 1010 including a computer-readable storage medium 1030 is shown. On the computer-readable storage medium 1030, a computer program 1020 may be stored that may cause the processing circuit 910 and entities and devices operatively coupled thereto (such as a communication interface 920 and a storage medium 930) to perform methods according to the embodiments described herein. The computer program 1020 and / or the computer program product 1010 may thus provide means for performing any of the steps disclosed herein.

[0081] exist Fig.10 In the example of , the computer program product 1010 is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-ray disc. The computer program product 1010 may also be implemented as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or an electrically erasable programmable read-only memory (EEPROM), and more specifically, as a non-volatile storage medium of a device in an external memory such as a USB (universal serial bus) memory or a flash memory (such as a compact flash). Therefore, although the computer program 1020 is schematically shown here as a track on the described optical disc, the computer program 1020 may be stored in any manner suitable for the computer program product 1010.

[0082] The inventive concept has mainly been described above with reference to a few embodiments. However, as readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept as defined by the appended patent claims.

Claims

1. A method for setting the frame rate of a camera (114), wherein: The camera (114) comprises a plurality of image sensors (116a:116K) arranged to capture images to be stitched into a panoramic image (500) of a scene (300, 400), and wherein each of the plurality of image sensors (116a:116K) has a respective optical axis (210a:210K), the method being performed by a controller (112), the method comprising: Obtain (S106) a corresponding angle (α) between the optical axis (210a:210K) of each of the plurality of image sensors (116a:116K) and a direction (D) k；k＝1…K ), a linear path structure (310, 410a, 410b, 510a, 510b) to be monitored by the camera (114) extending through the scene (300, 400) along the direction (D); According to the angle (α k；k＝1…K ), dividing (S108) the plurality of image sensors (116a:116K) into at least two groups (820a:820d), wherein all the angles (α k；k＝1…K ) within a continuous angular interval in the continuous angular intervals (810a:810e); and A frame rate is set (S110) for each of the at least two groups (820a:820d), wherein the frame rate decreases from the group (820b) corresponding to the highest continuous angular interval (810b, 810e) to the group (820a, 820c) corresponding to the lowest continuous angular interval (810a, 810c).

2. The method according to claim 1, wherein: The camera (114) has a camera axis (220) that is an average of the optical axes (210a:210K) of all the image sensors (116a:116K), wherein the frame rate of any given image sensor (116K) in different groups of the plurality of image sensors (116a:116K) decreases as the difference between the optical axis (210k) of any given image sensor (116k) and the camera axis (220) increases.

3. The method according to claim 1, wherein: The panoramic image (500) is composed of at least two images captured from different image sensors (116a: 116K) at the same point in time.

4. The method according to claim 1, wherein: The frame rate of a first group among the at least two groups (820a:820d) is a multiple of the frame rate of a second group among the at least two groups (820a:820d).

5. The method according to claim 4, wherein: The images captured by the image sensor (116a:116K) in the first group of the at least two groups (820a:820d) are captured at a first set of time opportunities, and the images captured by the image sensor (116a:116K) in the second group of the at least two groups (820a:820d) are captured at a subset of the first set of time opportunities.

6. The method according to claim 1, wherein: The continuous angle intervals (810a:810e) are composed of a plurality of threshold angle values ​​(β j；j＝1…J ) are separated, and wherein the number of groups (820a:820d) into which the plurality of image sensors (116a:116K) are divided depends on the threshold angle value (β j；j＝1…J ) 7. The method according to claim 1, wherein: The linear path structure (310, 410a, 410b, 510a, 510b) is a vehicle road, a sidewalk, an office corridor, a railway track, an airport taxiway, an airport runway, a ship channel, a part of a sports field or a runway of a sports field.

8. The method according to claim 1, wherein: The camera (114) has a camera axis (220), and wherein the camera axis (220) is parallel or perpendicular to the direction (D) along which the linear path structure (310, 410a, 410b, 510a, 510b) extends through the scene (300, 400).

9. The method according to claim 1, wherein: The frame rate takes a value from a set of predefined frame rate values.

10. The method according to claim 1, wherein: The method further comprises: The presence of the straight path structure (310, 410a, 410b, 510a, 510b) and / or the movement of the camera (114) relative to the straight path structure (310, 410a, 410b, 510a, 510b) is detected (S102) by performing object detection and recognition in the image, and wherein, in response thereto, the frame rate is set.

11. The method according to claim 1, wherein: The method further comprises: A user input is received (S104) to set a frame rate of the camera (114), and / or to reduce computational requirements of the camera (114), and wherein, in response thereto, the frame rate is set.

12. A method according to any preceding claim, wherein: The plurality of image sensors (116a: 116K) collectively span a field of view between 120° and 185°.

13. A controller (112) for setting a frame rate of a camera (114), wherein: The camera (114) includes a plurality of image sensors (116a:116K) arranged to capture images to be stitched into a panoramic image (500) of a scene (300, 400), and wherein each of the plurality of image sensors (116a:116K) has a respective optical axis (210a:210K), and the controller (112) includes a processing circuit (910) configured to cause the controller (112) to perform the following operations: Obtaining a corresponding angle (α) between the optical axis (210a:210K) of each of the plurality of image sensors (116a:116K) and a direction (D) k；k＝1…K ), a linear path structure (310, 410a, 410b, 510a, 510b) to be monitored by the camera (114) extending through the scene (300, 400) along the direction (D); According to the angle (α k；k＝1…K ), dividing the plurality of image sensors (116a:116K) into at least two groups (820a:820d), wherein all the angles (α k；k＝1…K ) within a continuous angular interval in the continuous angular intervals (810a:810e); and A frame rate is set for each of the at least two groups (820a:820d), wherein the frame rate decreases from the group (820b) corresponding to the highest continuous angular interval (810b, 810e) to the group (820a, 820c) corresponding to the lowest continuous angular interval (810a, 810c).

14. A system (110) for setting a frame rate, the system (110) comprising a controller (112) according to claim 13 and a camera (114) arranged as claimed in claim 13.

15. A computer-readable storage medium storing a computer program (1020) for setting a frame rate of a camera (114), wherein: The camera (114) comprises a plurality of image sensors (116a:116K) arranged to capture images to be stitched into a panoramic image (500) of a scene (300, 400), and wherein each of the plurality of image sensors (116a:116K) has a respective optical axis (210a:210K), and the computer program comprises computer code which, when executed on a processing circuit (910) of a controller (112), causes the controller (112) to: Obtain (S106) a corresponding angle (α) between the optical axis (210a:210K) of each of the plurality of image sensors (116a:116K) and a direction (D) k；k＝1…K ), a linear path structure (310, 410a, 410b, 510a, 510b) to be monitored by the camera (114) extending through the scene (300, 400) along the direction (D); According to the angle (α k；k＝1…K ), dividing (S108) the plurality of image sensors (116a:116K) into at least two groups (820a:820d), wherein all the angles (α k；k＝1…K ) within a continuous angular interval in the continuous angular intervals (810a:810e); and A frame rate is set (S110) for each of the at least two groups (820a:820d), wherein the frame rate decreases from the group (820b) corresponding to the highest continuous angular interval (810b, 810e) to the group (820a, 820c) corresponding to the lowest continuous angular interval (810a, 810c).

Images