Object position determination in frames of a video stream

Abstract

An object localization method comprises: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, deciding whether the determination of the position of at least one object (11, 13, 15, 17, 19) in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame.

Description

Technical Field

[0001] The present invention generally relates to a method, an object locator, a computer program, a computer program product and a user equipment for object position determination in a frame of a video stream. Background Art

[0002] Augmented reality (AR) is a direct or indirect live view of a physical, real-world environment whose objects are augmented (i.e., perceptually enriched) by computer-generated perceptual information. The superimposed perceptual information can be constructive, i.e., in addition to the natural environment, or destructive, i.e., obscuring the natural environment.

[0003] More and more AR applications have been developed for user devices (e.g., smartphones and tablet computers) to overlay virtual objects on real-world views. The core technical challenges in these applications are:

[0004] 1) Identifying real-world objects and their locations on the screen, often referred to in the art as object detection (OD) or object recognition;

[0005] 2) tracking objects of interest, often denoted in the art as object tracking (OT); and

[0006] 3) Augmenting the scene with artificial objects, labels, or other types of perceptual information.

[0007] Previously, some of the best solutions in the field of object detection were considered to be based on Deformable Part Models (DPM) with Histogram of Oriented Gradients (HOG) features. In the past few years, more accurate solutions based on Convolutional Neural Network (CNN) techniques have been considered the state of the art in the field of object detection. These solutions detect objects in a given frame or picture of a video stream, but require a lot of processing power to operate in real time. Therefore, CNNs are usually run on servers equipped with modern Graphics Processing Units (GPUs) with large amounts of memory.

[0008] In some AR applications, object detection needs to be run in real time on a portable user device. A typical example is an industrial AR application, which can be, for example, a support tool for technicians to repair complex hardware systems. Then, the portable user device (e.g., in the form of a handheld device or a head-mounted device) includes a camera for capturing video that is input to the object detection. If the camera of such a portable user device changes its position, object detection needs to be run in almost every frame of the video stream in order to find the current position of the object in the scene. However, due to the processing complexity of object detection and the limited processing power and power supply of the portable user device, it is often impossible to run object detection in every frame.

[0009] Traditionally, this problem has been solved by not running object detection on every frame, but running object detection periodically and instead tracking the detected objects between consecutive object detection runs. However, object tracking is generally less accurate than object detection and objects can be easily lost. Furthermore, object tracking cannot handle occlusions of tracked objects or detect new objects entering the scene. Furthermore, for example, if the scene is static, running object detection periodically is not computationally efficient, as object tracking can easily handle such static scenes. Another problem with running object detection periodically is that if new objects enter the scene between scheduled object detection runs, these objects will not be visualized in a timely manner.

[0010] Therefore, there is a need for more efficient object position determination suitable for implementation in portable user devices. Summary of the invention

[0011] A general object is to provide an object position determination suitable for implementation in a portable user device.

[0012] This and other objects are achieved by the aspects of the present invention and the embodiments disclosed herein.

[0013] An aspect of the present invention relates to an object localization method. The method comprises: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, deciding whether determination of a position of at least one object in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object detected in a reference frame.

[0014] Another aspect of the invention relates to an object locator, the object locator comprising processing circuitry and memory, the memory comprising instructions executable by the processing circuitry, the processing circuitry being operable to: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, determine whether determination of a position of at least one object in the at least one frame is based on object detection applied to the at least one frame or based on a transformation of a position of the at least one object detected in a reference frame.

[0015] Another aspect of the invention relates to a user device comprising an object locator according to the above and a camera configured to record a video and generate a video stream.

[0016] Yet another aspect of the present invention relates to a computer program comprising instructions which, when executed by at least one processing circuit, cause the at least one processing circuit to: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, decide whether determination of a position of at least one object in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object detected in the reference frame.

[0017] Another aspect of the present invention relates to a computer program product having stored thereon a computer program comprising instructions which, when executed on a processing circuit, cause the processing circuit to: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, decide whether determination of a position of at least one object in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object detected in the reference frame.

[0018] The present invention provides a multi-mode technique for determining the position of an object in a frame of a video stream. The multi-mode technique complements an object detection mode with a transform mode in which the position of an object in a reference frame is transformed or projected to a position in a current frame. According to the present invention, the computational complexity of determining the position of an object in a frame is reduced by the transform mode, thereby enabling implementation in a portable user device with limited computational and power resources. The multi-mode technique also enables visualization of perceptual information of fully or partially occluded objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The embodiments and further objects and advantages thereof may best be understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1 is an overview of a client-server architecture with object detection in the server;

[0021] Figure 2 is an overview of a user device including an object detector;

[0022] Figure 3 schematically showing a scene including a frame of a video stream of an object whose position is to be determined;

[0023] Figure 4 Schematic diagram showing the use of bounding boxes determined in object detection for augmentation Figure 3 Scenes;

[0024] Figure 5 schematically illustrates a scene of subsequent frames of a video stream after camera rotation and augmentation with bounding boxes determined in object detection;

[0025] Figure 6 Schematically illustrating the use of bounding boxes determined based on transformation operations for enhancement Figure 5 Scenes in;

[0026] Fig. 7A and Figure 7B Schematically shows the partial occlusion of the object and the use of transformation-based operations ( Fig. 7A ) or based on object detection ( Figure 7B ) determining a bounding box for augmenting the scene in subsequent frames of the video stream;

[0027] Figure 8 is a flow chart showing an object positioning method according to an embodiment;

[0028] Fig. 9 is a diagram showing a Figure 8 A flowchart of additional optional steps of the method shown;

[0029] Fig.10 is a diagram showing a method according to another embodiment Figure 8 A flowchart of additional optional steps of the method shown;

[0030] Fig.11 is a flowchart illustrating an object positioning method according to various embodiments;

[0031] Fig.12 is a diagram showing a Figure 8 A flowchart of additional optional steps of the method shown;

[0032] Fig.13 is a diagram showing a Figure 8 A flowchart of additional optional steps of the method shown;

[0033] Fig.14 is a diagram showing a Figure 8 A flowchart of additional optional steps of the method shown;

[0034] Fig.15 is a block diagram of an object locator according to an embodiment;

[0035] Fig.16 is a block diagram of an object locator according to another embodiment;

[0036] Fig.17 is a block diagram of an object locator according to another embodiment;

[0037] Fig.18 The implementation of the embodiment based on a computer program is schematically shown;

[0038] Fig.19 is a block diagram of an object locator according to another embodiment;

[0039] Fig. 20 Schematically illustrates the distributed implementation among network devices;

[0040] Fig.21 is a schematic diagram of an example of a wireless communication system having one or more cloud-based network devices according to an embodiment;

[0041] Fig. 22 is a schematic diagram illustrating an example of a telecommunications network connected to a host computer via an intermediary network according to some embodiments; and

[0042] Fig.23 is a schematic diagram illustrating an example of a host computer communicating with a user device via a base station over a partially wireless connection in accordance with some embodiments. DETAILED DESCRIPTION

[0043] Throughout the drawings, the same reference numerals are used for similar or corresponding elements.

[0044] Generally, unless clearly given and / or different meanings are implied from the context in which the term is used, all terms used in this article will be interpreted according to their common meaning in the relevant technical field. Unless otherwise clearly stated, all references to one / one / described element, equipment, assembly, device, step, etc. should be openly interpreted as referring to at least one example in element, equipment, assembly, device, step, etc. Unless a step must be clearly described as after or before another step and / or a step must be after or before another step implicitly, the steps of any method disclosed herein do not have to be performed in the exact order disclosed. Where applicable, any feature of any embodiment disclosed herein may be applied to any other embodiment. Similarly, any advantage of any embodiment may be applicable to any other embodiment, and vice versa. By the following description, other purposes, features and advantages of the attached embodiments will be apparent.

[0045] The present invention generally relates to a method, an object locator, a computer program, a computer program product and a user equipment for object position determination in a frame of a video stream.

[0046] Figure 1 The user device-server architecture for augmented reality (AR) is shown in FIG. Figure 1The user device 1 represented by the portable wireless user device 1 in FIG. 1 comprises or is connected to a camera 2 for capturing video and recording a video stream. Pictures or video frames (referred to herein as frames for simplicity) of the video stream are then sent from the user device 1 to an object detection (OD) server 5. The frame transmission may involve sending the video stream to the OD server 5, i.e., substantially all frames of the video stream are sent to the OD server 5. In an alternative embodiment, individual, typically time-stamped frames are sent to the OD server 5 for object detection.

[0047] The OD server 5 includes an object detector 4 for performing object detection on a received frame or at least a portion thereof. The object detection involves detecting an object in a processed frame and determining information about the detected object, including an object position representation, a detection probability, and an object type. The object position representation is commonly referred to in the art as a bounding box, defining an area of ​​the processed frame or an area within the processed frame. The detection probability represents the likelihood that an area of ​​the frame or an area within the frame defined by the object position representation includes an object. The object type defines the type or category of the detected object, such as a car, a pedestrian, a house, etc.

[0048] This so-called detection information (i.e. object position representation, detection probability and object type) is returned to the user device 1 together with an indication of which frame the object detection has been performed for (e.g. in terms of the timestamp of the relevant frame). The user device 1 then uses the detection information to enhance the video presented on the screen.

[0049] The OD server 5 can access an offline trained convolutional neural network (CNN) based object detector and a modern graphics processing unit (GPU) with a large amount of memory. Such a CNN typically includes tens of millions of parameters trained offline on a large annotated dataset, such as PASCAL VOC (Everingham et al., "The PASCAL Visual Object Classes (VOC) challenge", International Journal of Computer Vision (2010) 88: 303–338) or ImageNet (Deng et al., "ImageNet: A large-scale hierarchical image database", in 2009 IEEE Conference on Computer Vision and Pattern Recognition (2009)).

[0050] Examples of CNN-based object detectors4 include Faster R-CNN (Ren et al., “Faster R-CNN: Towards real-time object detection with region proposal networks”, IEEE Transactions on Pattern Analysis and Machine Intelligence (2017) 39(6): 1137– 1149), SSD (Liu et al., “SSD: Single shot multibox detector”, Proceedings of theEuropean Conference on Computer Vision (ECCV) (2016)), and YOLO9000 (Redmon and Farhadi, “YOLO9000: Better, faster, stronger”, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)).

[0051] In another implementation example, Figure 2 As shown, an object detector 4 is implemented in the user device 1. Therefore, the user device 1 then includes both a camera 2 for capturing video and generating a video stream and an object detector 4 for processing frames of the video stream for the purpose of detecting objects therein. Therefore, this implementation relaxes the need to send frames to a remote OD server 5 and receive detection information therefrom, but instead, it is necessary to implement an object detector 4 including an offline trained object detection model used by the object detector 4 at the user device 1.

[0052] Augmented reality finds more and more applications in portable user devices 1. A typical example is an industrial AR application, in which AR constitutes a support tool for technicians repairing complex hardware systems. In such a scenario, object detection should be run in real time, usually on the portable user device 1, which imposes restrictions on the complexity of the object detector 4. In most cases, the portability of the user device 1 and thus the portability of the camera 2 leads to a lot of movement when the technician is engaged in repairing the hardware system. However, also in these cases, the objects in the video should still be accurately detected and visualized. If the camera 2 changes its position relative to the hardware system being filmed, object detection should usually be run in each frame of the video stream in order to detect and classify objects in the current scene. However, this is usually not possible due to the complexity of object detection and the battery limitations of the portable user device 1.

[0053] The present invention solves the above-mentioned disadvantages when implementing AR applications in the portable user device 1 by adaptively switching between object detection mode and transformation mode (also called projection mode). This allows the AR application to run in real time in the portable user device 1 and to be able to visualize the position of the object in real time.

[0054] Figure 8 is a flow chart showing an object localization method according to an embodiment. The method comprises step S1, which is performed for at least one frame of a video stream, as schematically shown by line L1. Step S1 comprises: for at least one frame and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, deciding whether the determination of the position of at least one object in the at least one frame is based on object detection applied to the at least one frame, or based on a transformation of the position of the at least one object detected in the reference frame.

[0055] Thus, according to the invention, the position of at least one object in a frame may be determined by applying object detection (so-called object detection mode) or by transforming an already determined position of the object in a previous frame of the video stream (so-called reference frame) to a position in the current frame (so-called transformation mode). The decision or selection between the object detection mode and the transformation mode is based on at least one parameter representing a change in the scene from the reference frame to the current frame.

[0056] Object detection as used in object detection mode is accurate, but computationally intensive and power-hungry. However, the computational complexity of the position transformation used in transformation mode is relatively low. Therefore, the present invention enables replacing computationally intensive object detection in many frames of a video stream by position transformation, thereby reducing the computational requirements and power consumption of implementing AR in a portable user device 1.

[0057] The reference frame is a previous frame of the video stream, and more preferably a previous frame in the video stream to which object detection has been applied. Thus, object detection applied to the reference frame detects at least one object in the reference frame and generates an object position representation for the at least one detected object, and typically also generates a detection probability and an object type for each detected object.

[0058] Figure 3 An example of a hardware system in the form of a baseband switch 10 comprising a plurality of connectors 11, 13, 15, 17, 19 is shown as an illustrative example of objects of a scene to be detected and enhanced by a portable user device. Figure 4 Shown is applied to Figure 3The results of object detection in the scene are visualized with bounding boxes 21, 23, 25, 27, 29 around the detected objects 11, 13, 15, 17, 19. Figure 4 The bounding boxes 21, 23, 25, 27, 29 shown are examples of object position representations generated in object detection applied to a reference frame. Thus, the object position representation defines a corresponding region or bounding box in the reference frame that includes an object having a likelihood represented by the detection probability and having a type represented by the object type.

[0059] Thus, in an embodiment, the object position representations generated by object detection are bounding boxes. Each bounding box represents four parameter values ​​that define an area of ​​a frame. In such an embodiment, Figure 8 Step S1 of the invention comprises deciding based on at least one parameter whether the determination of the bounding box defining the area in the at least one frame is based on object detection applied to the at least one frame or on a transformation of a bounding box in a reference frame.

[0060] The bounding box may be in the form of a vector, for example, which defines the coordinates of the region and the size of the region. can be any coordinates that allow identification of the location of the region in the frame. The coordinates can, for example, represent the center of the region or represent one of the corners of the region. As an illustrative but non-limiting example, the size of the region can be determined by the width of the region and height Therefore, in an embodiment, the bounding box can be In an alternative embodiment, the bounding box may include the coordinates of the relative corners of the region, i.e., .

[0061] Object detection models and algorithms traditionally used in object detection in frames of a video stream output a bounding box, a detection probability, and an object type for each detected object, as mentioned earlier in this article. A bounding box is in most cases a rectangle or square defined by the four parameters described above. This is useful in imaging scenes such as Figure 5Restrictions may be imposed when detecting objects in the case of the rotation shown. The bounding box may then not align with any rotated objects detected in the scene. An advantage of the transform mode is that the transformations applied to the bounding box of an object in the reference frame can involve various operations, such as rotation, rescaling, translation, etc. This means that even if the bounding box defines a rectangular area in the reference frame, the transformed bounding box in the current frame does not necessarily need to define a rectangular area when the transform mode is applied. In sharp contrast, the transformed bounding box may define a quadrilateral area in the current frame. Illustrative but non-limiting examples of such quadrilateral areas include trapezoids, trapezoids (trapezoides), isosceles trapezoids (isosceles trapezoids), parallelograms, rhombuses, rhombuses, kites, rectangles, and squares. This is in Figure 6 It is schematically shown in Figure 4 The rectangle and secondary bounding boxes 21, 23, 25, 27, 29 have been transformed into Figure 6 Diamond shaped bounding boxes 21, 23, 25, 27, 29 are shown.

[0062] Therefore, in an embodiment, Figure 8 Step S1 in comprises: deciding based on at least one parameter whether to determine a bounding box defining a rectangular area in the at least one frame based on object detection applied to the at least one frame, or to determine a bounding box defining a quadrilateral area in the at least one frame based on a transformation of a bounding box in a reference frame.

[0063] The object detection used in the object detection mode can be based on any object detection algorithm implemented in the object detector. For example, object detection can be in the form of sliding window object detection, such as Viola and Jones, "Rapid Object Detection using a Boosted Cascade of Simple Features", in Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, Kauai, HI, US; or Fischler and Bolles, "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography, Communications of the ACM (1981) 24 (6): 381-395. Alternatively, object detection can be in the form of CNN-based object detection, such as the previously mentioned Faster R-CNN, SSD or YOLO9000.

[0064] You can do this by Figure 2 The object detector 4 implemented in the user device 1 shown in the figure is used to run the object detection used in the object detection mode. In another embodiment, the object detection is performed as in the example Figure 1 The object detection server 5 shown in the figure is implemented in a remote object detector 4. In such an embodiment, the user device 1 sends frames that should be subjected to object detection to the remote object detector 4, and receives object detection information, i.e., an object position representation, from it, and typically also receives detection probabilities and object types as previously disclosed herein. Running object detection remotely has the advantage of utilizing powerful GPUs (e.g., in the OD server 5), thereby using computationally intensive object detection models and algorithms to run at an acceptable frame rate. The disadvantage is that frames need to be sent to the remote object detector 4, and detection information returned from it, which can prolong the delivery of detection information to the user device 1 in the event of network congestion and heavy network load.

[0065] Figure 5 The rotation of the scene due to the movement of the portable user device 1 used to photograph the baseband switch 10 is shown. Figure 5shows the results of object detection applied to a rotated scene. A significant limitation of object detection is that the bounding boxes determined during object detection are rectangular and therefore not very effective when dealing with rotated scenes. Figure 5 In , only one object 15 is correctly detected and visualized with a bounding box 25. Figure 6 Shown with Figure 5 The same rotated scene, but when instead of applying object detection in object detection mode, by changing (see Figure 4 ) The positions of objects 11, 13, 15, 17, and 19 in the reference frame are transformed into Figure 6 The positions of objects 11, 13, 15, 17, and 19 are determined by the positions in the current frame. Figure 6 As shown, the transform mode can efficiently handle scene rotations and still visualize bounding boxes that have been transformed (e.g., rotated) relative to their positions in the reference frame 21, 23, 25, 27, 29.

[0066] Another significant advantage of using the transform mode is that it can handle occlusions by objects 13, 15, 17, such as Fig. 7A As shown. Fig. 7A In the example, another object 30 completely occludes the object with reference numeral 15 and partially occludes objects 13, 17. However, by changing the positions of these objects 13, 15, 17 Figure 4 The reference frame in the image is transformed into the current frame, and the bounding boxes 23, 25, 27 for these objects 13, 15, 17 can still be correctly visualized. This is not possible when using object detection mode or object tracking using existing techniques, such as Figure 7B Therefore, in Figure 7B In the figure, only objects 11 and 19 are correctly detected, while only parts of objects 13 and 17 are detected. Object 15 cannot be detected because it is hidden by object 30. The figure also shows bounding boxes 21, 23, 27, 29, 32 for the detected objects 11, 13, 17, 19, 30.

[0067] Fig. 9 is a diagram showing a Figure 8 Flowchart of additional optional steps of the method in step S10. The embodiment comprises estimating or determining a transformation matrix based on reference key points derived from the reference frame and key points derived from at least one frame. The transformation matrix defines a transformation of a position in the reference frame to a position in at least one frame. The method then proceeds to Figure 8In this embodiment, step S1 comprises: deciding based on at least one parameter whether the determination of the position of at least one object in at least one frame is based on object detection applied to at least one frame or based on a transformation of the position of at least one object detected in a reference frame using a transformation matrix.

[0068] Transformation Matrix Defines the position or orientation in a reference frame To the position or location in the current frame The transformation of .

[0069] According to different embodiments, various types of transformation matrices may be estimated in step S10. In a typical example, the transformation matrix defines a geometric transformation of a position between frames. A geometric transformation is a function whose domain and range are a set of points. Most commonly, both the domain and range of a geometric transformation are R 2 or both R 3 Geometric transformations can be 1-1 functions, i.e., they have an inverse function. Illustrative but non-limiting examples of geometric transformations include: affine transformations, which are functions between affine spaces, preserving points, lines, and planes, thereby maintaining parallelism; projective transformations, which are functions between projective spaces, preserving collinearity; and rotation-translation transformations.

[0070] In step S10, a transformation matrix is ​​estimated based on key points derived from a reference frame and from at least one frame. These key points are very unique points or features that can be identified and preferably tracked frame by frame in a video stream. Therefore, so-called reference key points are derived from a reference frame in a video stream. In a specific embodiment, the reference key points are extracted from a reference frame or identified in a reference frame. For example, the Shi-Tomasi algorithm (Shi and Tomasi, "Good Features to Track", in 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, CVPR94, Seattle, WA, US) can be used to identify reference key points in a reference frame.

[0071] The corresponding or matched key points are also derived from at least one frame. In an embodiment, the reference key points identified in the reference frame are tracked or followed in subsequent frames of the video stream until the current frame is reached. The key points can be selected based on various key point, feature or object tracking algorithms (e.g., the Lucas-Kanade optical flow algorithm (Lucas and Kanade, "An Iterative Image Registration Technique with an Application to Stereo Vision", in Proceedings of the 7 th international joint conference on Artificial intelligence (1981) 2:674-679, Vancouver, Canada) to perform tracking. In another embodiment, the keypoint identification algorithm applied to the reference frame can be reapplied, but then applied to the current frame to identify keypoints corresponding to reference keypoints in the reference frame.

[0072] As used herein, a matched or corresponding keypoint refers to a keypoint or feature in a reference frame that is the same as in the current frame. For example, the upper left corner of a box identified as a reference keypoint in the reference frame matches and corresponds to the upper left corner of the same box in the current frame, even though the position of the box may have changed from the reference frame to the current frame.

[0073] The transformation matrix may be estimated in step S10 based on the reference key points derived from the reference frame and the key points derived from at least one frame. Various matrix estimation methods may be used in step S10. For example, the elements of the transformation matrix may be estimated by means of least squares estimation (LSE). As an example, assuming Matched keypoints are derived from the reference frame and the current frame:

[0074]

[0075] as well as

[0076]

[0077] in, represents the x and y coordinates of the reference keypoint in the reference frame, and represents the x and y coordinates of the matched keypoints in the current frame, . Estimation of the transformation matrix can then consist of finding the optimal transformation of the form:

[0078] ,

[0079] The transformation matrix is . Get the transformation matrix The LSE solution is given by Given. Other algorithms and methods for estimating the transformation matrix from two sets (preferably two balanced sets, i.e., the number of key points in both sets is the same) can be used. An illustrative but non-limiting example of such another algorithm or method is RANSAC (Fischler and Bolles, "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography, Communications of the ACM (1981) 24(6): 381-395).

[0080] The transformation matrix then references the position in the frame Transform to the position in the current frame :

[0081]

[0082] In an embodiment, step S1 comprises deciding, based on at least one parameter derived from a transformation matrix, whether the determination of the position of at least one object in at least one frame is based on object detection applied to at least one frame, or on a transformation of the position of at least one object detected in a reference frame using a transformation matrix.

[0083] Therefore, in this embodiment, at least one parameter derived from the transformation matrix is ​​used as a parameter representing a change between a scene represented by at least one frame and a scene represented by a reference frame.

[0084] In an embodiment, the transformation matrix is ​​parameterized as:

[0085]

[0086] in

[0087]

[0088] here , are the horizontal and vertical scaling factors, is the rotation angle, and , is the horizontal and vertical translation.

[0089] Any of these parameters (i.e., scaling factor, rotation angle, and translation), or any combination of these parameters, can be used as Figure 8 The determination of the position of the at least one object in step S1 is based on whether the decision is based on object detection or on transformation.

[0090] Instead of or as an alternative to using at least one parameter derived from the transformation matrix, another parameter or other parameters representing a change between the scene represented by the at least one frame and the scene represented by the reference frame may be used as Figure 8 The basis for the decision in step S1. Fig.10 As shown in step S20, at least one parameter may be received from at least one sensor of the user device. In this embodiment, the user device 1 includes one or more sensors 2, 3, which can be used to generate or determine at least one parameter. Exemplary but non-limiting examples of such sensors 2, 3 include accelerometers, magnetometers, gyroscopes, and cameras, see Figure 1 and Figure 2 .

[0091] For example, operating systems for wireless communication devices (e.g., smartphones) including Android and iOS provide an application programming interface (API) to obtain the approximate rotation angle of the wireless communication device. These operating systems also provide access to raw data from various sensors (e.g., cameras, accelerometers, magnetometers, and gyroscopes) that can be used to estimate the position of the wireless communication device, and thus estimate the translation of the wireless communication device.

[0092] For example, the function getDefaultSensor(SENSOR_TYPE_ROTATION_VECTOR) reports the orientation of a wireless communication device running Android relative to an east-north-up coordinate frame. It is typically obtained by integrating the accelerometer, gyroscope, and magnetometer readings. For more information, see https: / / source.android.com / devices / sensors / sensor-types#rotation_vector Accordingly, the CMAttitude class provides the orientation of a wireless communication device running iOS, see https: / / developer.appe.com / documentation / coremotion / cmattitude .

[0093] There is also a trend for operating systems running in wireless communication devices (such as smartphones) to include simultaneous localization and mapping (SLAM) capabilities. SLAM includes algorithms that estimate position and orientation from both the camera and other sensors in the wireless communication device. For example, Android supports the ARCore library (https: / / developers.google.com / ar / reference / java / com / google / ar / core / Camera#getDisplayOrientedPose()), while iOS supports the ARKit library (https: / / developers.google.com / ar / reference / java / com / google / ar / core / Camera#getDisplayOrientedPose()). https: / / developer.apple.com / documentation / arkit / arcamera ).

[0094] Thus, given a position and orientation of a user device as obtained from at least one sensor, at least one parameter representing a change between a scene represented by at least one frame and a scene represented by a reference frame may be calculated, for example by calculating a scene rotation, a scene translation and / or a scene zoom.

[0095] Fig.11 is a flow chart showing an object positioning method according to various embodiments. In an embodiment, as Figure 8 Step S1 shown includes: Fig.11 Steps S31, S33, S35 and S36 are shown. In this embodiment, step S31 includes: if any rotation of the scene represented by at least one frame relative to the scene represented by the reference frame exceeds a threshold rotation, determining the position of at least one object in the at least one frame based on a transformation of the position of the at least one object detected in the reference frame.

[0096] In an embodiment, the method further comprises an optional step S30 comprising comparing the rotation of the scene with a threshold rotation. If the rotation of the scene exceeds the threshold rotation, the method proceeds to step S31.

[0097] Thus, if the scene represented by the current frame has been rotated more than a threshold relative to the scene represented by the reference frame, the position of at least one object in the current frame is determined according to a so-called transformation mode (i.e., a transformation based on the position of at least one object detected in the reference frame).

[0098] Figure 6 shows the scene relative to Figure 4 The rotation of the reference frame shown. Figure 6 As is clear from the example, the transformation applied to the bounding box 21, 23, 25, 27, 29 around the object 11, 13, 15, 17, 19 in step S31 may rotate and optionally translate the bounding box 21, 23, 25, 27, 29 so that it appears around the object 11, 13, 15, 17, 19 even after the scene has been rotated. Figure 5 If the object detection mode is used as shown in Figure 1, it cannot effectively handle this scene rotation. Figure 6 As shown, the transform mode can efficiently handle scene rotations and still visualize bounding boxes that have been transformed (e.g., rotated) relative to their positions in the reference frame 21, 23, 25, 27, 29.

[0099] If the rotation of the scene does not exceed a threshold rotation, such as verified in optional step S30, then if any reduction of the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold reduction, in step S33 a position of at least one object in the at least one frame is determined based on object detection applied to the at least one frame.

[0100] In an embodiment, the method further comprises an optional step S32, which comprises comparing the reduction of the scene with a threshold reduction. If the reduction of the scene exceeds the threshold reduction, the method proceeds to step S33.

[0101] Thus, if the scene represented by the current frame represents a scaled-down version of the scene represented by the reference frame, the position of at least one object in the current frame is determined according to the object detection mode (ie based on object detection applied to the current frame).

[0102] The reason for using object detection mode instead of transformation mode in case of large or severe zoom out (over a threshold zoom out) is that by zooming out, there is a high probability that new objects enter the scene and where these objects do not appear in the scene represented by the reference frame. Therefore, for these new objects entering the scene, there is no corresponding object in the scene represented by the reference frame. A typical example is having a reference frame in which the camera is zoomed in. Figure 3 The left part of the baseband switcher 10 in the scene is shown in FIG. 1 . In this case, the only object appearing in the scene will be object 11. If the camera then zooms out to capture the entire baseband switcher 10, new objects 13, 15, 17, 19 will appear in the scene. The position transformation between scenes applied in the transformation mode is only valid for objects that are already present and detected in the scene represented by the reference frame. This means that for severe zoom-out operations, which are most likely to introduce new objects, object detection should instead be used in step S33 in order to determine the position of at least one object in the current frame.

[0103] If the reduction of the scene does not exceed a threshold reduction, such as verified in optional step S32, then if any translation of the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold translation, in step S35, the position of at least one object in the at least one frame is determined based on object detection applied to the at least one frame.

[0104] In an embodiment, the method further comprises an optional step S34, which comprises comparing the translation of the scene with a threshold translation. If the translation of the scene exceeds the threshold translation, the method proceeds to step S35.

[0105] The translation of the scene may be a translation of the scene in the x direction, a translation of the scene in the y direction, or any translation that is independent of direction. For example, assume that the upper left corner of the baseband switch 10 corresponds to pixel (2, 9) in the reference frame and corresponds to pixel (25, 17) in the current frame. In this case, the translation of the scene in the x direction is 25-2 = 23 pixels, the translation of the scene in the y direction is 17-9 = 8 pixels, and a typical translation may be, for example, pixels.

[0106] In case of large translations, the reasons for using object detection mode instead of transform mode are basically the same as for zooming out, namely, there is a risk of new objects entering the current frame that were not present or detected in the reference frame.

[0107] If the translation of the scene does not exceed a threshold translation, such as verified in optional step S34, then in step S36 the position of the at least one object in the at least one frame is determined based on a transformation of the position of the at least one object detected in the reference frame.

[0108] In other words, in an embodiment, the object detection mode is used only in cases of severe zooming and translation, while the transform mode is used in cases of severe rotation and in all other cases where the object detection mode is not used.

[0109] The order of comparison in optional steps S30, S32 and S34 may be changed, for example, in any one of the following orders: S30, S34 and S32; S32, S30 and S34; S32, S34 and S30; S34, S30 and S32; or S34, S32 and S30.

[0110] Fig.12 is a diagram showing a Figure 8 Flow chart of additional optional steps of the method shown. This embodiment includes comparing at least one parameter with a corresponding threshold value. The method then continues Figure 8 In this embodiment, step S1 comprises: deciding based on the comparison whether the determination of the position of the at least one object in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object detected in a reference frame.

[0111] like Fig.11As shown, step S40 may be performed according to any one of steps S30, S32 and S34.

[0112] In an embodiment, the at least one parameter comprises a rotation angle In this embodiment, the method includes: if , then in Fig.11 In step S31, the position of the at least one object in the at least one frame is determined based on the transformation of the position of the at least one object detected in the reference frame, otherwise in step S33 or S35, the position of the at least one object in the at least one frame is determined based on the object detection applied to the at least one frame. In this embodiment, is a threshold, such as the threshold rotation mentioned previously.

[0113] In an alternative or additional embodiment, the at least one parameter comprises a horizontal scaling factor and the vertical scaling factor In this embodiment, the method includes: if , then in step S33, the position of at least one object in the at least one frame is determined based on the object detection applied to the at least one frame, otherwise in step S36, the position of at least one object in the at least one frame is determined based on the transformation of the position of the at least one object detected in the reference frame. In this embodiment, is the threshold value.

[0114] In this embodiment, low values ​​of the scaling factor represent severe zooming out.

[0115] In an alternative or additional embodiment, the at least one parameter includes a horizontal translation and vertical translation In this embodiment, the method includes: if , then in step S35, the position of at least one object in the at least one frame is determined based on the object detection applied to the at least one frame, otherwise in step S36, the position of at least one object in the at least one frame is determined based on the transformation of the position of the at least one object detected in the reference frame. In this embodiment, is a threshold, such as the threshold shift mentioned previously.

[0116] Therefore, as described above, any one of the above parameter examples can be used alone when deciding or selecting whether to use object detection mode or transformation mode. In another embodiment, at least two of the above parameters can be used to decide or select whether to use object detection mode or transformation mode, such as rotation and reduction, rotation and translation, or reduction and translation, or all three parameters can be used to decide or select whether to use object detection mode or transformation mode.

[0117] In this latter case, at least one parameter includes a horizontal scaling factor , vertical scaling factor , rotation angle , Horizontal translation and vertical translation The method then comprises: if , then in step S31, the position of at least one object in at least one frame is determined based on the transformation of the position of at least one object detected in the reference frame. , the method comprises: if or , then in step S33 or S35, the position of at least one object in the at least one frame is determined based on the object detection applied to the at least one frame, otherwise in step S36, the position of at least one object in the at least one frame is determined based on the transformation of the position of at least one object detected in the reference frame.

[0118] In an embodiment, the method comprises: Fig.13 The additional steps S50 and S51 are shown. Step S50 comprises converting a time parameter representing a time period from a reference frame to at least one frame in the video stream. With threshold For comparison. , the method proceeds to step S51. Step S51 comprises: determining the position of at least one object in the at least one frame based on object detection applied to the at least one frame. However, if , the method continues Figure 8 Step S1 in .

[0119] In another embodiment, the time parameter is used together with the scaling factor and the translation when deciding or selecting whether to use the object detection mode or the transformation mode. In this embodiment, the method includes: if , then in step S31, the position of at least one object in at least one frame is determined based on the transformation of the position of at least one object detected in the reference frame. , the method comprises: if or or , then in step S33 or S35, the position of at least one object in the at least one frame is determined based on the object detection applied to the at least one frame, otherwise in step S36, the position of at least one object in the at least one frame is determined based on the transformation of the position of at least one object detected in the reference frame.

[0120] Therefore, this embodiment introduces an initial check or criterion before deciding whether the determination of the position of at least one object is to be performed according to the object detection mode or according to the transformation mode. This initial check verifies that the reference frame is still recent, i.e. not too much time has passed from the reference frame in the video stream to the current frame in the video stream. For example, if the reference frame is at frame number 5 in the video stream, and the current frame is at frame number 305, then the reference frame may not be a good reference for any object in the current frame, because the scene is likely to have changed a lot during these 300 frames. In this case, it is better to apply object detection to the current frame instead in order to determine the position of any object.

[0121] The time parameter can express the time in seconds, for example In another example, the time parameter represents the number of frames, e.g. These examples are equivalent because given the frame number and the frame rate of the video stream, the difference in frames can be converted to time, for example, using a frame rate of 30 fps, 300 frames represents 10 seconds of video. It is also possible to convert time to the number of frames using the frame rate.

[0122] In an embodiment, if, for example, Fig.11 In step S33 or S35 of the video stream, object detection is applied to at least one frame, and the at least one frame can be used as a reference frame for a subsequent frame or a next frame of the video stream.

[0123] In an embodiment, the reference frame is the most recent frame in the video stream to which object detection has been applied to determine the position of at least one object. Thus, if the current frame of the video stream has frame number j, in this embodiment the reference frame is the frame to which object detection has been applied and which has frame number jk, where k is as low a number as possible. Although it is generally preferred to use the most recent frame to which object detection has been applied as the reference frame for any subsequent frames in the video stream, embodiments are not limited thereto. This means that, as an alternative, another frame to which object detection has been applied may be used as the reference frame instead of the most recent previous reference frame, i.e., a frame with frame number jl is used as the reference frame instead of a frame with frame number jk, where l>k, and the frames with frame numbers jk and jl both contain at least one object detected using object detection, e.g. Fig.11 In step S35 or step S33 in the latter embodiment, the above-mentioned time parameter is used in combination. This temporal parameter ensures that no outdated previous frame is used as a reference frame for the current frame.

[0124] Fig.14 It shows Figure 8Flow chart of additional optional steps of the method shown. Step S2 includes: based on the position of at least one object in at least one frame, using the perceptual information to enhance the at least one frame. The enhanced at least one frame can then be output for display on a screen of a user device.

[0125] Thus, by using the position of any object determined according to embodiments based on the object detection mode or the transformation mode, at least one frame may be enhanced with perceptual information based on the position of the object.

[0126] In particular embodiments, the type of perceptual information used to enhance the at least one frame may be selected based on the type of object determined in the object detection.

[0127] As used herein, perception information refers to any information or data that can be used to enhance a scene. Non-limiting but illustrative examples of such perception information include the name of a detected building, the name of a detected person, etc.

[0128] Examples of sensory information can be Figures 3 to 7B Bounding boxes 21, 23, 25, 27, 29 are shown around the objects. The visualized bounding boxes 21, 23, 25, 27, 29 may optionally be supplemented with information or identifiers of the objects 11, 13, 15, 17, 19 enclosed by the bounding boxes 21, 23, 25, 27, 29. The information or identifier may, for example, identify the name or type of a connector in the baseband switch 10.

[0129] In a specific embodiment, the proposed method includes four main steps:

[0130] A. Object Detection: Run object detection in the reference frame to obtain bounding boxes for objects of interest.

[0131] B. Projection: Use sparse point-to-point matching to find the transformation matrix H that projects the reference frame (e.g., the last frame subjected to object detection step A) to the current frame. Use the transformation matrix to project the bounding box detected in the reference frame to the current frame.

[0132] C. Frame evaluation: Evaluate the parameters in the transformation matrix or from the sensor and decide when to switch between the object detection step A and the projection step B.

[0133] D. Visualization: Draw the detected or projected bounding boxes in the current frame and render them on the screen.

[0134] In this particular embodiment, the object detection step can be performed by sliding window object detection or CNN-based object detection as illustrative examples. Object detection takes a frame as input and outputs a bounding box of the detected object. The projection step is based on the estimation of a transformation matrix. In this particular embodiment, the transformation matrix is ​​obtained by first extracting very unique points (i.e., key points) from a reference frame and tracking them in subsequent frames. Next, the transformation matrix is ​​estimated based on the matched key points extracted from the reference frame and tracked in the current frame.

[0135] In an embodiment, the four main steps of the proposed algorithm can be implemented according to the following operations:

[0136] 1. Get a reference frame from the video source (such as the camera of the user's device) as the current frame.

[0137] 2. Run object detection on the current frame to find objects and bounding boxes.

[0138] 3. For example, use the Shi-Tomasi algorithm to extract reference key points from the current frame.

[0139] 4. Draw the bounding box and render the augmented frame on the screen of the user’s device.

[0140] 5. For each subsequent frame, do:

[0141] a. For example, use the Lucas-Kanade algorithm to find the current position of the key point.

[0142] b. Use the reference keypoints from step 3 and their current positions to estimate the transformation matrix, for example using LSE or RANSAC.

[0143] c. Calculate based on transformation matrix and / or sensor data , , , and .

[0144] d. If , then go to step 5f, otherwise continue to step 5e. Here, is a predefined angle threshold, for example, a value of 5°.

[0145] e. If , then a severe shrinkage / severe shift has occurred or the time has passed If the allowed threshold has been exceeded, go to step 2, otherwise proceed to step 5f. Here, and are thresholds. The values ​​of these thresholds are preferably proportional to the size of the smallest object detected. is a predetermined interval, for example, equal to 3s.

[0146] f. Transform or project the bounding box from step 2 into the current frame using the transformation matrix.

[0147] g. Draw the bounding box and render the augmented frame on the user device’s screen.

[0148] In another embodiment of this particular algorithm, the elapsed time of execution and its threshold The check between is performed as a separate step between step 5c and step 5d.

[0149] Another aspect of the embodiment relates to an object locator, the object locator comprising processing circuitry and memory, the memory comprising instructions executable by the processing circuitry, the processing circuitry operative to: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, determine whether determination of a position of at least one object in the at least one frame is based on object detection applied to the at least one frame or based on a transformation of a position of the at least one object detected in a reference frame.

[0150] In an embodiment, the processing circuit is operative to estimate a transformation matrix based on reference key points derived from a reference frame and key points derived from at least one frame. The transformation matrix defines a transformation of a position in the reference frame to a position in the at least one frame. In this embodiment, the processing circuit is further operative to decide based on at least one parameter whether the determination of the position of at least one object in the at least one frame is based on object detection applied to the at least one frame or based on a transformation of the position of the at least one object detected in the reference frame using the transformation matrix.

[0151] In an embodiment, the processing circuit is also operative to determine, based on at least one parameter derived from the transformation matrix, whether the determination of the position of at least one object in at least one frame is based on object detection applied to at least one frame, or based on a transformation of the position of at least one object detected in a reference frame using the transformation matrix.

[0152] In an embodiment, the processing circuit is operative to receive the at least one parameter from at least one sensor of a user device.

[0153] In an embodiment, the processing circuit is operative to: determine the position of the at least one object in the at least one frame based on a transformation of the position of the at least one object detected in the reference frame if any rotation of the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold rotation, otherwise, determine the position of the at least one object in the at least one frame based on object detection applied to the at least one frame if any reduction of the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold reduction, otherwise, determine the position of the at least one object in the at least one frame based on object detection applied to the at least one frame if any translation of the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold translation, otherwise, determine the position of the at least one object in the at least one frame based on a transformation of the position of the at least one object detected in the reference frame.

[0154] In an embodiment, the processing circuit is operable to compare the at least one parameter with a corresponding threshold value. In this embodiment, the processing circuit is further operable to decide based on the comparison whether the determination of the position of the at least one object in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object detected in the reference frame.

[0155] In an embodiment, the at least one parameter comprises a rotation angle In this embodiment, the processing circuit is operable to: if , the position of the at least one object in the at least one frame is determined based on a transformation of the position of the at least one object detected in the reference frame, otherwise, the position of the at least one object in the at least one frame is determined based on object detection applied to the at least one frame. is the threshold value.

[0156] In an embodiment, the at least one parameter comprises a horizontal scaling factor and the vertical scaling factor In this embodiment, the processing circuit is operable to: if , then determining the position of the at least one object in the at least one frame based on the object detection applied to the at least one frame, otherwise, determining the position of the at least one object in the at least one frame based on the transformation of the position of the at least one object detected in the reference frame. is the threshold value.

[0157] In an embodiment, the at least one parameter comprises a horizontal translation and vertical translation In this embodiment, the processing circuit is operable to: if , then determining the position of the at least one object in the at least one frame based on the object detection applied to the at least one frame, otherwise, determining the position of the at least one object in the at least one frame based on the transformation of the position of the at least one object detected in the reference frame. is the threshold value.

[0158] In an embodiment, the at least one parameter comprises a horizontal scaling factor , vertical scaling factor , rotation angle , Horizontal translation and vertical translation In this embodiment, the processing circuit is operable to: if , then the position of the at least one object in the at least one frame is determined based on the transformation of the position of the at least one object detected in the reference frame. Otherwise, if or , then determining the position of the at least one object in the at least one frame based on the object detection applied to the at least one frame, otherwise, determining the position of the at least one object in the at least one frame based on the transformation of the position of the at least one object detected in the reference frame.

[0159] In an embodiment, the processing circuit is operative to: convert a time parameter representing a time period from the reference frame to the at least one frame in the video stream into With threshold In this embodiment, the processing circuit is used to: if , then determining the position of the at least one object in the at least one frame based on the object detection applied to the at least one frame, otherwise, deciding based on the at least one parameter whether the determination of the position of the at least one object in the at least one frame is based on the object detection applied to the at least one frame or on a transformation of the position of the at least one object detected in the reference frame.

[0160] In an embodiment, the processing circuit is operative in this embodiment to: if , then the position of the at least one object in the at least one frame is determined based on the transformation of the position of the at least one object detected in the reference frame, otherwise, if or or , then determining the position of the at least one object in the at least one frame based on the object detection applied to the at least one frame, otherwise, determining the position of the at least one object in the at least one frame based on the transformation of the position of the at least one object detected in the reference frame.

[0161] In an embodiment, the processing circuit is operative to decide based on the at least one parameter whether the determination of the bounding box defining the region in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the bounding box in the reference frame.

[0162] In an embodiment, the processing circuit is operative to decide, based on the at least one parameter, whether to determine a bounding box defining a rectangular area in the at least one frame based on object detection applied to the at least one frame, or to determine a bounding box defining a quadrilateral area in the at least one frame based on a transformation of a bounding box in the reference frame.

[0163] In an embodiment, the processing circuit is operative to enhance the at least one frame using perceptual information based on a position of the at least one object in the at least one frame.

[0164] Another aspect of the embodiment relates to an object locator, which is configured to: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, decide whether determination of a position of at least one object in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object detected in a reference frame.

[0165] It will be understood that the methods, method steps and devices, device functions described herein may be implemented, combined and rearranged in various ways.

[0166] For example, embodiments may be implemented in hardware, or in software executed by suitable processing circuitry, or a combination thereof.

[0167] The steps, functions, processes, modules and / or blocks described herein may be implemented in hardware using any conventional techniques, such as using discrete circuit or integrated circuit technology, including both general purpose electronic circuits and application specific circuits.

[0168] Alternatively, or in addition, at least some of the steps, functions, processes, modules and / or blocks described herein may be implemented in software, such as a computer program executed by a suitable processing circuit (eg, one or more processors or processing units).

[0169] Examples of processing circuitry include, but are not limited to, one or more microprocessors, one or more digital signal processors (DSPs), one or more central processing units (CPUs), video acceleration hardware, and / or any suitable programmable logic circuitry, such as one or more field programmable gate arrays (FPGAs) or one or more programmable logic controllers (PLCs).

[0170] It should also be understood that the general processing capabilities of any conventional device or unit implementing the proposed technology can be reused. For example, existing software can also be reused by reprogramming the existing software or by adding new software components.

[0171] Fig.15 is a schematic block diagram illustrating an example of an object locator 100 according to an embodiment. In this particular example, the object locator 100 comprises a processing circuit 101 (eg a processor) and a memory 102. The memory 102 comprises instructions executable by the processing circuit 101.

[0172] Optionally, the object locator 100 may further include a communication circuit. Fig.15 102. The I / O unit 103 may include functionality for wired and / or wireless communications with other devices, servers, and / or network nodes in a wired or wireless communications network. In a specific example, the I / O unit 103 may be based on a radio circuit for communicating with one or more other nodes (including sending and / or receiving information). The I / O unit 103 may be interconnected to the processing circuit 101 and / or the memory 102. As an example, the I / O unit 103 may include any of the following: a receiver, a transmitter, a transceiver, an I / O circuit, an input port, and / or an output port.

[0173] Fig.16 1 is a schematic block diagram showing an object locator 110 implemented based on a hardware circuit according to an embodiment. Specific examples of suitable hardware circuits include: one or more suitably configurable or possibly reconfigurable electronic circuits, for example, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs); or any other hardware logic, such as circuits based on interconnected discrete logic gates and / or flip-flops to perform dedicated functions together with suitable registers (REGs) and / or memory units (MEMs).

[0174] Fig.171 is a schematic block diagram showing yet another example of an object locator based on a combination of processing circuit(s) 122, 123 and hardware circuits 124, 125, in conjunction with a suitable memory unit 121. Thus, the overall functionality is divided between programmed software for execution on one or more processing circuits 122, 123 and one or more preconfigured or possibly reconfigurable hardware circuits 124, 125. The actual hardware-software division may be determined by the system designer based on numerous factors, including processing speed, implementation cost and other requirements.

[0175] Fig.18 It is a computer program-based implementation of the object locator 200 according to an embodiment. In this specific example, at least some of the steps, functions, processes, modules and / or boxes described herein are implemented with a computer program 240, wherein the computer program 240 is loaded into the memory 220 for execution by a processing circuit including one or more processing circuits 210. The processing circuit (one or more) 210 and the memory 220 are interconnected to each other to achieve normal software execution. The optional I / O unit 230 can also be interconnected to the processing circuit (one or more) 210 and / or the memory 220 to achieve input and / or output of relevant data (such as frames and detection information).

[0176] The term "processing circuitry" should be interpreted in a general sense as any circuit, system or device capable of executing program code or computer program instructions to perform specific processing, determination or computing tasks.

[0177] Thus, the processing circuit 210 is configured to, when executing the computer program 240 , perform well-defined processing tasks such as those described herein.

[0178] The processing circuitry is not necessarily dedicated to performing only the above-mentioned steps, functions, processes and / or blocks, but may also perform other tasks.

[0179] In an embodiment, the computer program 240 comprises instructions which, when executed by the at least one processing circuit 210, cause the at least one processing circuit 210 to: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, decide whether determination of a position of at least one object in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object detected in the reference frame.

[0180] The proposed technology also provides a carrier 250, also referred to as a computer program product, comprising the computer program 240. The carrier 250 is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal or a computer readable storage medium.

[0181] As an example, the software or computer program 240 is stored on a computer-readable storage medium (particularly a non-volatile medium) such as the memory 220. The computer-readable medium may include one or more removable or non-removable memory devices, including (but not limited to): read-only memory (ROM), random access memory (RAM), compact disk (CD), digital versatile disk (DVD), Blu-ray disk, universal serial bus (USB) memory, hard disk drive (HDD) storage device, flash memory, magnetic tape, or any other conventional memory device. The computer program 240 may thus be loaded into the operating memory 220 for execution by the processing circuit 210.

[0182] The computer program product 250 has stored thereon a computer program 240 comprising instructions which, when executed on the processing circuit 201, cause the processing circuit to: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, decide whether the determination of the position of at least one object in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object detected in the reference frame.

[0183] When executed by one or more processors, the flowcharts presented herein can be considered as computer flowcharts. The corresponding device can be defined as a set of functional modules, where each step performed by the processor corresponds to a functional module. In this case, the functional module is implemented as a computer program running on the processor.

[0184] Therefore, the computer program resident in the memory may be organized into suitable functional modules, which are configured to, when executed by the processor, perform at least part of the steps and / or tasks described herein.

[0185] Fig.19 is a block diagram of an object locator 130. The object locator 130 comprises a decision module 131 for deciding, for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, whether determination of a position of at least one object in the at least one frame is based on object detection applied to the at least one frame or on a transformation of a position of the at least one object detected in a reference frame.

[0186] Another aspect relates to a user device 1, see Figure 1 and Figure 2 , which comprises an object locator according to the invention, for example in combination with Figures 15 to 19 Any one of the descriptions in . In an embodiment, the user device 1 further comprises or is connected to a camera 2 configured to record a video and generate a video stream. In an embodiment, the user device 1 further comprises or is connected to at least one sensor 2, 3 configured to generate at least one parameter.

[0187] In an embodiment, the user device is selected from the group consisting of a computer, a laptop, a smart phone, a mobile phone, a tablet computer, a multimedia player, a set-top box and a game console.

[0188] It is also becoming increasingly common to provide computing services (hardware and / or software) in network devices such as network nodes and / or servers, where resources are provided as services to remote locations over a network. As an example, this means that the functions described herein can be distributed or relocated to one or more separate physical nodes or servers. The functions can be relocated or distributed to one or more co-working physical and / or virtual machines that can be located at separate physical nodes, i.e., in the so-called cloud. This is sometimes also referred to as cloud computing, which is a model that supports on-demand network access to a pool of configurable computing resources such as networks, servers, storage devices, applications, and general or customized services, anytime, anywhere.

[0189] There are different forms of virtualization that are useful in this context, including one or more of the following:

[0190] Consolidating network functions as virtualized software running on custom or general-purpose hardware. This is sometimes called network function virtualization.

[0191] Co-locating one or more application stacks (including operating systems) running on separate hardware onto a single hardware platform. This is sometimes referred to as system virtualization or platform virtualization.

[0192] Co-location of hardware and / or software resources with the goal of obtaining improved system resource utilization using some advanced domain-level scheduling and coordination techniques. This is sometimes referred to as resource virtualization, or centralized and coordinated resource pooling.

[0193] While it is often desirable to centralize functionality into a so-called common data center, in other scenarios it may be advantageous to actually distribute functionality over different parts of the network.

[0194] A network device can generally be considered as an electronic device that is communicatively connected to other electronic devices in a network. As an example, a network device can be implemented in hardware, software, or a combination thereof. For example, a network device can be a dedicated network device or a general-purpose network device or a mixture thereof.

[0195] A dedicated network device may use custom processing circuitry and a proprietary operating system (OS) to execute software to provide one or more of the features or functionality disclosed herein.

[0196] A general purpose network device may use a common off-the-shelf (COTS) processor and a standard OS to execute software configured to provide one or more of the features or functions disclosed herein.

[0197] As an example, a dedicated network device may include hardware, a physical network interface (NI), and a non-transitory machine-readable storage medium on which software is stored, wherein the hardware includes processing or computing resources, which typically includes a set of one or more processors, and the physical network interface (NI) is sometimes referred to as a physical port. The physical NI can be regarded as hardware in a network device for making a network connection, such as wirelessly through a wireless network interface controller (WNIC) or by plugging a cable into a physical port connected to a network interface controller (NIC). During operation, the software can be executed by the hardware to instantiate a set of one or more software instances. Each software instance and the portion of the hardware that executes the software instance can form a separate virtual network element.

[0198] As another example, a general network device may include, for example, hardware including a set of one or more processors (typically COTS processors) and a NIC and a non-transitory machine-readable storage medium on which software is stored. During operation, the processor executes software to instantiate one or more sets of one or more applications. Although one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization, such as represented by a virtualization layer and a software container. For example, one such alternative embodiment implements virtualization at the operating system level, in which case the virtualization layer represents an operating system kernel that allows the creation of multiple software containers or a pad (shim) executed on the base operating system, each of which can be used to execute one of the application sets. In an example embodiment, each software container (also referred to as a virtualization engine, a virtual private server, or a jail) is a user space instance (typically a virtual storage space). These user space instances can be separated from each other and from the kernel space that executes the operating system. Unless explicitly allowed, a set of applications running in a given user space cannot access the memory of other processes. Another such alternative embodiment implements full virtualization, in which case: 1) the virtualization layer represents a hypervisor (sometimes called a virtual machine monitor (VMM)), or a hypervisor executing on top of a host operating system; and, 2) each software container represents a tightly isolated form of software container called a virtual machine that is executed by the hypervisor and may include a guest operating system.

[0199] According to yet another embodiment, a hybrid network device is provided that includes both custom processing circuitry / proprietary OS and COTS processor / standard OS in the network device (e.g., a card or circuit board within the network device). In certain embodiments of such a hybrid network device, a platform virtual machine (VM) (e.g., a VM that implements the functionality of a dedicated network device) can provide paravirtualization to hardware present in the hybrid network device.

[0200] Fig. 20 is a schematic diagram showing an example of how functions can be distributed or divided between different network devices in general. In this example, there are at least two separate but interconnected network devices 300, 310, which can divide different functions or parts of the same function between network devices 300 and 310. There may be additional network devices 320 that are part of such a distributed implementation. The network devices 300, 310, 320 can be part of the same wireless or wired communication system, or one or more network devices can be so-called cloud-based network devices located outside the wireless or wired communication system.

[0201] As used herein, the term "network device" may refer to any device connected to a communication network, including but not limited to devices in access networks, core networks, and similar network structures. The term network device may also include cloud-based network devices.

[0202] Fig.21 is a schematic diagram showing an example of a wireless communication system including a radio access network (RAN) 41 and a core network 42 and an optional operation support system (OSS) 43 cooperating with one or more cloud-based network devices 300. The figure also shows a user equipment 1 connected to the RAN 41 and capable of wireless communication with a RAN node 40, such as a network node, a base station, a Node B (NB), an evolved Node B (eNB), a next generation Node B (gNB), etc.

[0203] exist Fig.21 The network device 300 shown in FIG. 3 as a cloud-based network device 300 may alternatively be implemented in conjunction with or at a RAN node 40 , for example.

[0204] Specifically, the proposed technology can be applied to specific applications and communication scenarios, including providing various services within wireless networks, including so-called over-the-top (OTT) services. For example, the proposed technology implements and / or includes the transmission and / or sending and / or receiving of relevant user data and / or control data in wireless communications.

[0205] In the following, reference will now be made to Fig. 22 and Fig.23 A collection of illustrative, non-limiting examples is described.

[0206] Fig. 22 is a schematic diagram illustrating an example of a telecommunications network connected to a host computer via an intermediary network according to some embodiments.

[0207] Reference Fig. 22According to an embodiment, a communication system includes a telecommunication network QQ410 (e.g., a cellular network of the 3GPP type), the telecommunication network QQ410 including an access network QQ411 (e.g., a radio access network) and a core network QQ414. The access network QQ411 includes a plurality of base stations QQ412a, QQ412b, QQ412c (e.g., NB, eNB, gNB or other types of wireless access points), each of which defines a corresponding coverage area QQ413a, QQ413b, QQ413c. Each base station QQ412a, QQ412b, QQ412c is connectable to the core network QQ414 via a wired or wireless connection QQ415. A first UE QQ491 located in the coverage area QQ413c is configured to be wirelessly connected to the corresponding base station QQ412c or to be paged by the corresponding base station QQ412c. A second UE QQ492 in the coverage area QQ413a is wirelessly connectable to the corresponding base station QQ412a. Although multiple UEs QQ491 , QQ492 are shown in this example, the disclosed embodiments are equally applicable to situations where only a UE is in the coverage area or a only UE is connected to the corresponding base station QQ412 .

[0208] The telecommunications network QQ410 itself is connected to a host computer QQ430, which may be implemented as hardware and / or software of a standalone server, a cloud-implemented server, a distributed server, or as a processing resource in a server cluster. The host computer QQ430 may be under the ownership or control of a service provider, or may be operated by or on behalf of a service provider. The connections QQ421 and QQ422 between the telecommunications network QQ410 and the host computer QQ430 may extend directly from the core network QQ414 to the host computer QQ430, or may be made via an optional intermediate network QQ420. The intermediate network QQ420 may be one or more of a public, private, or bearer network; the intermediate network QQ420 (if present) may be a backbone network or the Internet; specifically, the intermediate network QQ420 may include two or more subnetworks (not shown).

[0209] Fig. 22The communication system as a whole implements a connection between the connected UE QQ491, QQ492 and the host computer QQ430. The connection can be described as an over-the-top (OTT) connection QQ450. The host computer QQ430 and the connected UE QQ491, QQ492 are configured to use the access network QQ411, the core network QQ414, any intermediate network QQ420 and possible other infrastructure (not shown) as intermediaries to transmit data and / or signaling via the OTT connection QQ450. The OTT connection QQ450 can be transparent in the sense that the participating communication devices through which the OTT connection QQ450 passes are not aware of the routing of the uplink and downlink communications. For example, the base station QQ412 may not be notified or may not need to be notified of the past routing of the incoming downlink communication with data originating from the host computer QQ430 to be forwarded (e.g., handed over) to the connected UE QQ491. Similarly, base station QQ412 need not be aware of the future routing of outgoing uplink communications originating from UE QQ491 to host computer QQ430.

[0210] Fig.23 is a schematic diagram illustrating an example of a host computer communicating with a user device via a base station over a partially wireless connection in accordance with some embodiments.

[0211] Now refer to Fig.23 To describe the example implementation of the UE, base station and host computer discussed in the previous paragraph according to the embodiment. In the communication system QQ500, the host computer QQ510 includes hardware QQ515, and the hardware QQ515 includes a communication interface QQ516, which is configured to establish and maintain a wired or wireless connection with the interface of different communication devices of the communication system QQ500. The host computer QQ510 also includes a processing circuit QQ518, which may have storage and / or processing capabilities. Specifically, the processing circuit QQ518 may include one or more programmable processors, application-specific integrated circuits, field programmable gate arrays, or a combination thereof (not shown) suitable for executing instructions. The host computer QQ510 also includes software QQ511, which is stored in the host computer QQ510 or can be accessed by the host computer QQ510 and can be executed by the processing circuit QQ518. The software QQ511 includes a host application QQ512. The host application QQ512 is operable to provide services to a remote user (e.g., UE QQ530) connected via an OTT connection QQ550 terminated at the UE QQ530 and the host computer QQ510. In providing services to the remote user, the host application QQ512 may provide user data sent using the OTT connection QQ550.

[0212] The communication system QQ500 also includes a base station QQ520 provided in the telecommunication system, the base station QQ520 including hardware QQ525 enabling it to communicate with the host computer QQ510 and with the UE QQ530. The hardware QQ525 may include: a communication interface QQ526 for establishing and maintaining a wired or wireless connection with the interface of different communication devices of the communication system QQ500; and a radio interface QQ527 for establishing and maintaining at least a connection with a network located in the coverage area ( Fig.23 The communication interface QQ526 may be configured to facilitate a connection QQ560 to the host computer QQ510. The connection QQ560 may be direct, or it may pass through the core network ( Fig.23 The base station QQ520 may be configured to communicate with the base station QQ520 via a plurality of intermediate networks (not shown) and / or via one or more intermediate networks external to the telecommunications system. In the illustrated embodiment, the hardware QQ525 of the base station QQ520 also includes processing circuitry QQ528, which may include one or more programmable processors, application specific integrated circuits, field programmable gate arrays, or combinations thereof (not shown) adapted to execute instructions. The base station QQ520 also has software QQ521 stored internally or accessible via an external connection.

[0213] The communication system QQ500 also includes the UE QQ530 already mentioned. The hardware QQ535 may include a radio interface QQ537, which is configured to establish and maintain a wireless connection QQ570 with a base station serving the coverage area where the UE QQ530 is currently located. The hardware QQ535 of the UE QQ530 also includes a processing circuit QQ538, which may include one or more programmable processors, application specific integrated circuits, field programmable gate arrays, or a combination thereof (not shown) suitable for executing instructions. The UE QQ530 also includes software QQ531, which is stored in the UE QQ530 or can be accessed by the UE QQ530 and can be executed by the processing circuit QQ538. The software QQ531 includes a client application QQ532. The client application QQ532 is operable to provide services to human or non-human users via the UE QQ530 with the support of the host computer QQ510. In the host computer QQ510, the executing host application QQ512 can communicate with the executing client application QQ532 via the OTT connection QQ550 terminated at the UE QQ530 and the host computer QQ510. When providing services to users, the client application QQ532 can receive request data from the host application QQ512 and provide user data in response to the request data. The OTT connection QQ550 can transmit both the request data and the user data. The client application QQ532 can interact with the user to generate the user data it provides.

[0214] Notice, Fig.23 The host computer QQ510, base station QQ520 and UE QQ530 shown can be respectively Fig. 22 The host computer QQ430, one of the base stations QQ412a, QQ412b, QQ412c and one of the UEs QQ491, QQ492 may be similar or identical. That is, the internal workings of these entities may be similar to or identical to those of the host computer QQ430, one of the base stations QQ412a, QQ412b, QQ412c and one of the UEs QQ491, QQ4 Fig.23 As shown, and independently, the surrounding network topology can be Fig. 22 network topology.

[0215] exist Fig.23 , an OTT connection QQ550 has been abstractly drawn to illustrate communications between a host computer QQ510 and a UE QQ530 via a base station QQ520, without explicitly mentioning any intermediate devices and the precise routing of messages via these devices. The network infrastructure may determine the routing, which may be configured to be hidden from the UE QQ530 or from the service provider operating the host computer QQ510, or from both. The network infrastructure may also make decisions to dynamically change the routing (e.g., based on load balancing considerations or reconfiguration of the network) while the OTT connection QQ550 is active.

[0216] The wireless connection QQ570 between the UE QQ530 and the base station QQ520 is in accordance with the teachings of the embodiments described throughout the present disclosure. One or more of the various embodiments improves the performance of OTT services provided to the UE QQ530 using the OTT connection QQ550, wherein the wireless connection QQ570 forms the last segment in the OTT connection QQ550.

[0217] For the purpose of monitoring the data rate, latency and other factors improved by one or more embodiments, a measurement process may be provided. There may also be an optional network function for reconfiguring the OTT connection QQ550 between the host computer QQ510 and the UE QQ530 in response to changes in the measurement results. The measurement process and / or network function for reconfiguring the OTT connection QQ550 may be implemented in software QQ511 and hardware QQ515 of the host computer QQ510 or in software QQ531 and hardware QQ535 of the UE QQ530 or in both. In an embodiment, a sensor (not shown) may be deployed in or associated with a communication device through which the OTT connection QQ550 passes; the sensor may participate in the measurement process by providing the value of the monitoring quantity exemplified above or providing the value of other physical quantities that the software QQ511, QQ531 can use to calculate or estimate the monitoring quantity. Reconfiguration of the OTT connection QQ550 may include message formats, retransmission settings, preferred routes, etc.; the reconfiguration need not affect the base station QQ520, and it may be unknown or imperceptible to the base station QQ520. Such processes and functions may be known and practiced in the art. In a particular embodiment, the measurement may involve proprietary UE signaling that facilitates the host computer QQ510 to measure throughput, propagation time, latency, etc. The measurement may be implemented as follows: the software QQ511 and QQ531 enable the use of the OTT connection QQ550 to send messages (specifically, empty messages or "fake" messages) while it monitors propagation time, errors, etc.

[0218] The embodiments described above will be understood as several illustrative examples of the present invention. It will be appreciated by those skilled in the art that various modifications, combinations and changes may be made to the embodiments without departing from the scope of the present invention. In particular, different partial solutions in different embodiments may be combined in other configurations where technically feasible. However, the scope of the present invention is defined by the appended claims.

Claims

1. An object positioning method, comprising the following steps: For at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, it is decided (S1) whether the determination of the position of at least one object (11, 13, 15, 17, 19) in the at least one frame is based on object detection applied to the at least one frame, or based on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame to the position in the at least one frame, wherein the change is caused by a camera configured to record the video and generate the video stream changing its position, and the at least one parameter is at least one of the following: a zoom factor, a rotation angle, or a translation of the scene.

2. The object positioning method according to claim 1, further comprising the following steps: A transformation matrix is ​​estimated (S10) based on reference key points derived from the reference frame and key points derived from the at least one frame, wherein the transformation matrix defines a transformation of a position in the reference frame to a position in the at least one frame, and wherein the decision step (S1) comprises: based on the at least one parameter, deciding (S1) whether the determination of the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame using the transformation matrix.

3. The object positioning method according to claim 2, wherein: The decision step (S1) comprises: based on the at least one parameter derived from the transformation matrix, deciding (S1) whether the determination of the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is based on object detection applied to the at least one frame, or based on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame using the transformation matrix.

4. The object positioning method according to any one of claims 1 to 3, further comprising the following steps: The at least one parameter is received (S20) from at least one sensor (2, 3) of the user equipment (1).

5. The object positioning method according to any one of claims 1 to 3, wherein: The decision step (S1) comprises: determining (S31) a position of the at least one object (11, 13, 15, 17, 19) in the at least one frame based on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame if any rotation of the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold rotation, otherwise determining (S33) a position of the at least one object (11, 13, 15, 17, 19) in the at least one frame based on object detection applied to the at least one frame if any reduction in the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold reduction, otherwise determining (S35) a position of the at least one object (11, 13, 15, 17, 19) in the at least one frame based on object detection applied to the at least one frame if any translation of the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold translation, otherwise The position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is determined (S36) based on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame.

6. The object positioning method according to any one of claims 1 to 3, further comprising the following steps: The at least one parameter is compared with a corresponding threshold value (S40), wherein the decision step (S1) comprises: deciding (S1) based on the comparison whether the determination of the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame.

7. The object positioning method according to claim 6, wherein: The at least one parameter includes a rotation angle , and wherein the decision step (S1) comprises: if , then based on the transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame, determining (S31) the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, otherwise, based on the object detection applied to the at least one frame, determining (S33, S35) the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, wherein is the threshold value.

8. The object positioning method according to claim 6, wherein: The at least one parameter comprises a horizontal scaling factor and the vertical scaling factor , and wherein the decision step (S1) comprises: if , then based on the object detection applied to the at least one frame, determining (S33) the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, otherwise, based on the transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame, determining (S36) the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, wherein is the threshold value.

9. The object positioning method according to claim 6, wherein: The at least one parameter includes a horizontal translation and vertical translation , and wherein the decision step (S1) comprises: if , then based on the object detection applied to the at least one frame, determining (S35) the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, otherwise, based on the transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame, determining (S36) the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, wherein is the threshold value.

10. The object positioning method according to claim 6, wherein: The at least one parameter comprises a horizontal scaling factor , vertical scaling factor , rotation angle , Horizontal translation and vertical translation , and wherein the decision step (S1) comprises: if , then based on the transformation of the position of the at least one object detected in the reference frame, determining (S31) the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, wherein is the threshold, otherwise if or , then determining (S33, S35) the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame based on the object detection applied to the at least one frame, wherein is the threshold, otherwise Based on the transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame, the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is determined (S36).

11. The object positioning method according to any one of claims 1 to 3 and 7 to 10, further comprising the following steps: A time parameter representing a time period from the reference frame to the at least one frame in the video stream With threshold Comparison is made (S50); and if , then based on the object detection applied to the at least one frame, determine (S51) the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, otherwise, decide (S1) based on the at least one parameter whether the determination of the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is based on the object detection applied to the at least one frame or on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame.

12. The object positioning method according to any one of claims 1 to 3 and 7 to 10, wherein: The decision step (S1) comprises: deciding (S1) based on the at least one parameter whether the determination of the bounding box (21, 23, 25, 27, 29) defining the area in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the bounding box (21, 23, 25, 27, 29) in the reference frame.

13. The object positioning method according to claim 12, wherein: The decision step (S1) comprises: deciding (S1) based on the at least one parameter whether to determine a bounding box (21, 23, 25, 27, 29) defining a rectangular area in the at least one frame based on object detection applied to the at least one frame, or to determine a bounding box (21, 23, 25, 27, 29) defining a quadrilateral area in the at least one frame based on a transformation of a bounding box (21, 23, 25, 27, 29) in the reference frame.

14. The object positioning method according to any one of claims 1 to 3, 7 to 10 and 13, further comprising the following steps: Based on the position of the at least one object in the at least one frame, the at least one frame is enhanced (S2) using perceptual information.

15. An object locator (100, 120), comprising: Processing circuit (101, 122); as well as A memory (102, 121) comprising instructions executable by the processing circuit (101, 122), wherein the processing circuit (101, 122) is operative to: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, determine whether the determination of the position of at least one object (11, 13, 15, 17, 19) in the at least one frame is based on object detection applied to the at least one frame or based on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame to the position in the at least one frame, wherein the change is caused by a camera configured to record video and generate a video stream changing its position, and the at least one parameter is at least one of the following: a zoom factor, a rotation angle, or a translation of the scene.

16. The object locator (100, 120) according to claim 15, wherein: The processing circuit (101, 122) is operable to: estimating a transformation matrix based on reference keypoints derived from the reference frame and keypoints derived from the at least one frame, wherein the transformation matrix defines a transformation of a position in the reference frame to a position in the at least one frame; and Based on the at least one parameter, it is decided whether the determination of the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is based on object detection applied to the at least one frame, or based on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame using the transformation matrix.

17. The object locator (100, 120) according to claim 16, wherein: The processing circuit (101, 122) is operable to decide, based on the at least one parameter derived from the transformation matrix, whether the determination of the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is based on object detection applied to the at least one frame, or based on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame using the transformation matrix.

18. The object locator (100, 120) according to any one of claims 15 to 17, wherein: The processing circuit (101, 122) is operable to: receive the at least one parameter from at least one sensor (2, 3) of the user equipment (1).

19. The object locator (100, 120) according to any one of claims 15 to 17, wherein: The processing circuit (101, 122) is operable to: determining a position of the at least one object (11, 13, 15, 17, 19) in the at least one frame based on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame if any rotation of the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold rotation, otherwise determining a position of the at least one object (11, 13, 15, 17, 19) in the at least one frame based on object detection applied to the at least one frame if any reduction in the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold reduction, otherwise determining a position of the at least one object (11, 13, 15, 17, 19) in the at least one frame based on object detection applied to the at least one frame if any translation of the scene represented by the at least one frame relative to the scene represented by the reference frame exceeds a threshold translation, otherwise The position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is determined based on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame.

20. The object locator (100, 120) according to any one of claims 15 to 17, wherein: The processing circuit (101, 122) is operable to: comparing the at least one parameter to a corresponding threshold value; as well as Based on the comparison, it is decided whether the determination of the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is based on object detection applied to the at least one frame or on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame.

21. The object locator (100, 120) of claim 20, wherein: The at least one parameter includes a rotation angle , and wherein the processing circuit (101, 122) is operable to: if , then based on the transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame, the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is determined, otherwise, based on the object detection applied to the at least one frame, the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is determined, wherein is the threshold value.

22. The object locator (100, 120) of claim 20, wherein: The at least one parameter comprises a horizontal scaling factor and the vertical scaling factor , and wherein the processing circuit (101, 122) is operable to: if , then based on the object detection applied to the at least one frame, determining the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, otherwise, determining the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame based on the transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame, wherein is the threshold value.

23. The object locator (100, 120) of claim 20, wherein: The at least one parameter includes a horizontal translation and vertical translation , and wherein the processing circuit (101, 122) is operable to: if , then based on the object detection applied to the at least one frame, determining the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, otherwise, determining the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame based on the transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame, wherein is the threshold value.

24. The object locator (100, 120) of claim 20, wherein: The at least one parameter comprises a horizontal scaling factor , vertical scaling factor , rotation angle , Horizontal translation and vertical translation , and wherein the processing circuit (101, 122) is operative to: if , then based on the transformation of the position of the at least one object detected in the reference frame, determining the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, wherein is the threshold, otherwise if or , then determining the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame based on the object detection applied to the at least one frame, wherein is the threshold, otherwise Based on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame, a position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is determined.

25. The object locator (100, 120) according to any one of claims 15 to 17 and 21 to 24, wherein The processing circuit (101, 122) is operable to: A time parameter representing a time period from the reference frame to the at least one frame in the video stream With threshold Make comparisons; and if , then based on the object detection applied to the at least one frame, determine the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame, otherwise, decide based on the at least one parameter whether the determination of the position of the at least one object (11, 13, 15, 17, 19) in the at least one frame is based on the object detection applied to the at least one frame or on a transformation of the position of the at least one object (11, 13, 15, 17, 19) detected in the reference frame.

26. The object locator (100, 120) according to any one of claims 15 to 17 and 21 to 24, wherein: The processing circuit (101, 122) is operable to decide, based on the at least one parameter, whether determination of a bounding box (21, 23, 25, 27, 29) defining a region in the at least one frame is based on object detection applied to the at least one frame or on a transformation of a bounding box (21, 23, 25, 27, 29) in the reference frame.

27. The object locator (100, 120) of claim 26, wherein: The processing circuit (101, 122) is operable to decide, based on the at least one parameter, whether to determine a bounding box (21, 23, 25, 27, 29) defining a rectangular area in the at least one frame based on object detection applied to the at least one frame, or to determine a bounding box (21, 23, 25, 27, 29) defining a quadrilateral area in the at least one frame based on a transformation of a bounding box (21, 23, 25, 27, 29) in the reference frame.

28. The object locator (100, 120) according to any one of claims 15 to 17, 21 to 24 and 27, wherein The processing circuit (101, 122) is operative to enhance the at least one frame using perceptual information based on a position of the at least one object in the at least one frame.

29. A user equipment (1), comprising: The object locator (100, 120) according to any one of claims 15 to 28; as well as The camera (2) is configured to record video and generate a video stream.

30. The user equipment (1) according to claim 29, further comprising: At least one sensor (2, 3) is configured to generate at least one parameter.

31. A computer readable storage medium having stored thereon a computer program (240) comprising instructions which, when executed on a processing circuit (210), cause the processing circuit (210) to: for at least one frame of a video stream and based on at least one parameter representing a change between a scene represented by the at least one frame and a scene represented by a reference frame of the video stream, decide whether a determination of a position of at least one object in the at least one frame is based on object detection applied to the at least one frame or on a transformation of a position of the at least one object detected in the reference frame to a position in the at least one frame, wherein The change is caused by a camera configured to record the video and generate the video stream changing its position, and the at least one parameter is at least one of: a zoom factor, a rotation angle, or a translation of the scene.

Images